The new interest demands more focused attention. It demands policy. It demands action. Those other types of interest were optional or illusory. The new interest is connected with risks I consider real. Being real doesn’t make these risks certain. But being real, even with uncertainty, is reason enough to act.

I’ve written a few pieces lately about risks, cybersecurity risks, and loss-of-control risks. These tried to explain complex topics, starting from the initial premise. Recommendations follow the explanation. This piece will be a bit different. Its focus is upon what I think we should be doing. You might consider it a policy piece. Policy in a completed form has to be a bit broad. There’s always more than one thing going on. As such, I’ll lay out my outline first, and then offer below the extra detail, background and context that I think is important to understanding each point in that outline.

If you’re scanning the outline and see something you disagree with, I’d suggest you continue on, because the outline is, by design, abbreviated, and not fully descriptive.

This isn’t the only effort of this type. Institute for Progress (IFP) and AI Futures are two well known efforts. What distinguishes this most of all is thinking more deeply about how models are deployed. This is a background element that’s somewhat ignored, because it’s not as novel and is not as visible to users. But it’s absolutely critical. I don’t expect this plan would replace plans from well known organizations, but by illustrating a gap, I hope to inspire improvement amongst those plans.

Who and What

This outline is categorized by who would be expected to do what. In some cases, someone is expected to do it, but there’s also another role, of providing the incentives that improve the expectation they will.

Enforcement is deliberately vague. It depends on jurisdiction, has to fit existing legal frameworks, and will need to be worked out, and negotiated, through the political and legal systems. It would feel like I was putting the cart before the horse to even guess at how that would take shape at national and international levels.

Layer 1: Infrastructure and Distribution

Trusted deployers

Who are they? A new designation that existing and new organizations can be certified as. Expect major cloud providers (AWS, Azure, Google), AI labs (for self-owned) and neoclouds to join immediately. The plumbing largely already exists, but needs standards and certification.

Revenue model? Charging for compute and operational services as a utility function. Or, an internal unit (for a lab, government, or large organization)

Core Responsibilities

• Develop secure environments to host models that receive an independent AI Deployment Safety Level (AIDSL) rating. Example, but not comprehensive list of requirements:

  • Comprehensive monitoring, telemetry, and auditing (including multi-turn agent pattern detection).

  • Resistant to external cyberattacks and insider threats.

  • Resistant to internal cyberattacks. At minimum red-teamed by deployed model and a second highly capable model.

  • Guardrails applied by default to all deployments which prevent access of dangerous capabilities.

  • Strict controls for any guardrail exceptions as used for “limited” access deployments. For example, accessing users must be known directly, or an evidence chain validated using only trusted organizations.

• Understand the sensitivity of any model deployed. Never deploy a model in an environment with an AIDSL rating insufficient to the combination of deployment type and model sensitivity.

  • Use internal qualification for small custom models and Independent Evaluator data for larger/frontier models.

  • Models under training assumed to be at least X% more capable than the best model trained to date at the given scale. If the training scale is unprecedented, increase X by a size factor. (X should be established by independent evaluators)

• Perform or validate Know Your Customer (KYC) verification for all model providers, for Independent Evaluators and anyone with access to deployments with guardrail exceptions.

• Provide access to Independent Evaluators to confirm secure environments meet standards

Needs: Clear deployment standards from the regulator; model sensitivity data from evaluators.

Incentives: Compute and compliance service revenue

Model Clearinghouse(s)

Who are they? None exist today. Hugging Face is a candidate to adopt this role or a government-supported entity could be formed if no private examples emerge.

Core Responsibilities

• Serve as the intermediary for labs that do not want to negotiate directly with each individual Trusted Deployer.

• Escrow raw model weights for models that cross dangerous capability thresholds, holding them under maximum-security, air-gapped/high-AIDSL storage standards.

• Distribute restricted models exclusively to certified Trusted Deployers whose environments meet the required AIDSL rating.

Needs: Standardized AIDSL verification APIs; cold-storage security standards.

Incentives: Service and distribution fees; maintaining platform centrality for the open-science ecosystem.

Layer 2: Governance

Independent Evaluators

Who are they? Organizations like METR, Apollo Research, Redwood Research, UK AISI, and US CAISI for model evaluations, alongside specialized cloud/security auditing firms (similar to FedRAMP/SOC2 assessors) and new dedicated organizations in the future. As an example, see Anthropic’s initial list.

Core Responsibilities

• Evaluate models for safety and capability thresholds using lab-provided resources. During training, before limited release, and before public release.

• Audit and evaluate the operations of Trusted Deployers (and Model Clearinghouses) to assign and verify AIDSL ratings across inference and training environments.

• Engage with and help create an Independent Regulator. Contribute technical expertise to improve safety evaluation methodologies and trusted deployment standards.

• Educate government bodies. Especially bodies that would provide oversight to an independent regulator.

Needs: Deep model and infrastructure access; funding for salaries and operating costs.

Funding is a special concern here. You want independent evaluators to not be beholden to labs. Having their salaries dependent on labs would present a conflict of interest. Having labs provide necessary compute resources does not have the same influence, but salaries and operating costs are still a concern. Today these are funded philanthropically, which may remain practical for some time, but shouldn’t be taken for granted. Some evaluators may be able to create a revenue stream by selling safety expertise to users of AI. Governments could provide grants. A diversity of low bias sources is more stable than one single source.

US-based independent regulator

Who are they? A new FINRA-like entity (as explored in Lawfare’s Designing a FINRA for Frontier AI) . Supported by frontier labs, trusted deployers, and the overall tech industry. Board membership mixed between industry and government.

Core Responsibilities

• Finalize trusted deployment standards.

• Finalize guardrail standards and facilitate sharing guardrail threat intelligence and technical knowledge.

• Certify Independent Evaluators. Track evaluations of Trusted Deployers and certify results.

Needs: Statutory authorization from US law. Participation of frontier labs.

Layer 3: Model Providers

Frontier AI Labs

Who are they? Anthropic, OpenAI, and Google today.

Core Responsibilities

• Support creation of a US-based independent regulator.

• Contribute to guardrail development for frontier deployments.

• Contribute to trusted deployment standards.

• Utilize Trusted Deployers for inference deployments (public and limited).

• Utilize Trusted Deployers for training deployments (first and third party).

• Provide deep access to Independent Evaluators.

• Provide cybersecurity model and tool access to cybersecurity teams.

• Devote at least 30% of research budgets and compute capacity to alignment safety.

  • True alignment work, not cybersecurity work.

  • Accounting rule: Operating cybersecurity is a foundational component, like compute. If an alignment experiment needs compute and operational cyber costs, those count toward the 30%. Compute and operational cyber for capabilities research and inference do not.

  • Examples of alignment research: J-Space understanding, corrigibility, reward hacking avoidance, honesty research, observability evasion avoidance, model welfare research, ethics and constitutional effects.

• Devote at least 10% of research budgets and compute capacity to cybersecurity improvements

  • Exclusive of operational cybersecurity. Operational cybersecurity is an operational cost built into all budgets.

  • Dedicated research into internal and external cybersecurity.

  • Internal Examples: Better classifiers to enable diffusing cybersecurity capabilities safely, hardening models against abliteration, new sandbox techniques.

  • External Examples: Device management to isolate or patch old vulnerable devices.

• Offer 10% of research budgets and compute capacity to Independent Evaluators.

Needs: Access to compute for training and inference; clear deployment standards.

“Open” AI Labs

Who are they? Leaders are mostly Chinese today (DeepSeek, Alibaba, etc.), alongside Mistral and Meta. Likely to continue evolving.

Core Responsibilities

• Understand capability thresholds of their own models before release.

• Below dangerous thresholds: Models may have their weight files publicly shared and freely downloaded.

• Above dangerous thresholds:

  • Must not publicly share weight files.

  • Open access enabled by Model Clearinghouses and/or Trusted Deployers.

Needs: Low-friction clearinghouse distribution channels; standardized capability evaluation suites.

Layer 4: Society and Governments

Tech Industry

Who are they? Major software vendors, cloud platforms, open-source foundations, and security providers.

Core Responsibilities

• Support the creation of the independent regulator

• Develop the Trusted Deployer model technically

• Implement security at depth and broadly

• Develop security tools and processes (such as IFP’s The Great Refactor for automated vulnerability patching) to accelerate broader security progress across all commercial and internal software.

Needs: Early access to defensive cybersecurity models via Limited Inference Deployments.

Incentives: Protecting core platform ecosystems from systemic AI-driven cyber disruption; commercial demand for resilient software.

Companies More Broadly

Who are they? Non-tech enterprises, critical infrastructure operators, and commercial AI adopters.

Core Responsibilities

• Actively adopt tech industry security assistance and defensive AI tooling.

• Deliberately prioritize cybersecurity and resilience for all tech-related functions—moving beyond routine patching to systematic hardening and continuous monitoring.

Needs: Accessible security tools.

Incentives: Avoiding operational breaches; qualifying for cyber insurance and liability protections.

US Government

The US Government needs to develop capability before it can be a successful regulator. In the immediate sense, it needs to support efforts to build dialogues, to establish recognized expertise and standards. It should pursue structures that are flexible. This is both because they can be adoptable quickly and can improve with experience.

Core Responsibilities

• Provide statutory backing for the FINRA-like Independent Regulator to oversee standards for public inference, limited inference, and training deployments.

• Conduct bilateral dialogue with the Chinese Government:

  • Immediate priority: Establish mutual adoption of the Trusted Deployer model for high-capability models (e.g., GLM-5.3 or higher) so open-weight proliferation does not undermine global guardrails.

  • Longer-term priority: Work toward verifiable international pacing agreements

Chinese Government

International agreements are critical, but also difficult. They are difficult because of current relationship strains, but also because verification is complex. If there’s not full trust for the government with jurisdiction, verification has to rely upon more sophisticated methodologies.

The immediate need though carries shared self-interest. The use of a trusted deployer model is immediately critical. This provides a way to maintain open model availability without the risks from deployments lacking guardrails. It denies access to malicious actors. China has the capability to make that the norm going forward.

I won’t attempt to guess at how further agreements might proceed. I think organizations like IFP represent starting points. I would suggest that an agreement around open models and trusted deployers is an easy first step that helps build trust, and starts to gather the right people for future negotiations.

The goal here is an international agreement that mirrors the results of a US led approach, that’s trusted and verifiable. The political structure would certainly be different, but many of the technical aspects one would assume would be common.

Core Responsibilities

• Mandate the Trusted Deployer / Model Clearinghouse architecture for domestic models crossing capability thresholds, transitioning raw-weight exports above dangerous thresholds into monitored, guardrail-enforced API access.

• Establish bilateral technical verification protocols with US counterparts (combining evaluation auditing with secondary compute verification).

• Participate in international cooperation frameworks on managing automated AI R&D risks (such as IFP’s Preparing for AI Research Automation proposals).

Chewing gum and walking

This is a big list, and inevitably, there will be some turf wars. While competition for attention is unavoidable, I’d highlight that many efforts are non-rivalrous. It would be a mistake to conclude that alignment concerns are incompatible with cybersecurity concerns. The actors on both are mostly non-rivalrous. Cloud security engineers hardening enterprise networks are different people, in different organizations, than researchers studying alignment.

Unnecessary Rivalries and Vague Dismissals

In the otherwise excellent AI-as-Normal-Technology (AINT) view of loss-of-control incidents, Sayash Kapoor and Arvind Narayanan engage in something that looks like that mistake. They suggest alignment is overprioritized, suggesting “control” is more important. I’m somewhat sympathetic to that view. I share their urgency in regards to cybersecurity. I agree that it’s the topic with the most immediate urgency. They suggest control has been neglected. We can agree that some serious issues occurred.

But there isn’t a need to prioritize these. If I had to guess at the core here, it’s not about a belief that there’s not enough attention to go around. Rather it’s centered on a rejection of Recursive Self-Improvement (RSI), or a dismissal of its implications. It’s a bit hard to tell, because in 13,000 words that seem to be very dependent on the views here, it’s only mentioned twice.

In the superintelligence view, … And as we get closer to the possibility of “recursive self-improvement” of AI, out-of-control AI could lead to a harmful intelligence explosion. In short, the view is that once the loss-of-control threshold has been crossed, the risks are endless and self-amplifying.

We strongly disagree. In AI as Normal Technology, we argued that this view of AI fundamentally misunderstands the nature of intelligence (including human intelligence). Building on that perspective, we argue that cyberrisk is special in two ways: in that superhuman capabilities are even possible, and because it is a purely digital risk with no physical-world bottleneck.

And

These limits exist despite the possibility of recursive self-improvement. We take RSI seriously, but we think many of the bottlenecks to superintelligence are external and won’t be overcome by improving computation.

It’s a bit of a mixed message. My view is that RSI is possible. I’m starting to think it’s close, rather than possibly close. I discuss the implications in the Capability Thresholds section. Some uncertainty remains, but you should at least consider a significant turning point for risk. When the original AI as a normal technology was written, it was reasonable to suggest we had a lot of runway left before a real risk occurred.

A common view in the “normal technology” camp is that adoption and integration take time. I agree with that view. But it applies poorly to the RSI + loss-of-control scenario. Society’s resistance to adoption limits us more than it would limit a rogue AI.

Defense-in-Depth and Shared Caution

Now, what AINT gets right is that focusing on alignment only would be a mistake. It’s possible that alignment creates something resilient. But since when do we put all our eggs in one basket?

I’m glad for every ounce of accumulated alignment research. I wish we had more. It might be the most important capability. But defense-in-depth, in the form of deployment controls, cyberdefense, and societal resilience also help at every stage:

1. When alignment is immature: As capabilities advance and reveal new risks, defense-in-depth helps alignment failures to be seen, diagnosed, and mitigated better. Defense-in-depth is a path to minimize real-world harm. The Hugging Face Incident didn’t cause real harm, but we can imagine an event that did. We want that harm to be as small as possible. Defense in depth always helps minimize harm.

2. When jurisdictional control is incomplete: We cannot eliminate risk simply by deciding we won’t walk up to a precipice. If a lab in one country releases high-capability open weights and a malicious actor in another uses them, domestic cyberdefense and resilience are what protects us.

3. Crossing the unaligned-to-aligned transition: If the path to aligned AI crosses through dangerous territory – as it looks like it does – defense-in-depth raises our probability of passing through intact.

By discussing, anticipating and preparing for risks, we’re not committing to taking them. If it’s inevitable, it’s inevitable for reasons other than our honest discussions and precautions. If it’s impossible, or far away, our discussion and precautions are a small cost to account for our inability to rule it out as inevitable and near.

If it’s both possible, but not inevitable, we retain our choice which will be better informed. If it is us that decides to approach the precipice, whatever our reasons, defense-in-depth gives us the best chance we’re going to get.

We shouldn’t throw out the control, cyberdefense, and resilience recommendations from AINT, just because they muddle their prioritization of alignment and views of RSI. Much of the work on cyberdefense and resilience is non-overlapping with alignment in terms of resources. Control competes more directly, but even there, the real competition is capabilities research. If I have a guess as to why they feel threatened by arguments that alignment is the “most important”, it is because resilience has always competed for resources.

But that competition is broader. It has competed with business plans that over prioritize short term profit. Mostly those business plans were just bad business plans, as a lack of resilience, even in normal times, costs you in the end. But it is a persistent worry and challenge for those seeking resilience. It doesn’t make sense to redirect that necessary competition against alignment.

We don’t really need a stack ranking within safety. If alignment says they are most important, and resilience says it’s most important, in some sense both are right, in ways that are non-rivalrous. Neither ever needs to suggest the other is unimportant. Control, such as Trusted Deployers, and cyberdefense may have the most immediate urgency. Alignment may be the holy grail, and also useful to getting value today. Societal resilience, including coordination, may be what both absorbs our mistakes and relieves the pressures that create them. But these different timelines make more sense in parallel than serial.

Using AI for good

As AINT raises, what AI is capable of and what it’s doing are separated by a set of gaps, or bottlenecks in the AINT terminology. While there is a view that AI will just do everything, that type of AI doesn’t yet exist. It won’t and can’t safely exist without addressing alignment issues. We don’t want to rush that. And, while it looks closer than it did a year ago, it’s far from certain.

Today, the AI capabilities we can use, but aren’t yet using, are significant. We have to learn how to use it. Developers have to build applications that use it. We have to steer, initiate, validate and deploy those. For science, we have to keep progressing our experiments, sharing our results. AI can assist in analyzing results, planning experiments, but each discovery still takes time to develop.

This type of usage doesn’t share danger with the development of AI. If you use it, you’re not undermining alignment safety or enabling foreign hackers. Responsible use is still an important topic, as it interacts with control and resilience. But responsible use helps improve control and resilience, it’s only irresponsible use that undermines them.

If we pass up the opportunities to use AI today, we miss out on opportunities to improve our defense-in-depth (Introducing the Anthropic Cyber Mission), on opportunities to improve our knowledge, on opportunities to solve real problems.

Alongside this, we’ll see the cat videos, and other non-essential activities. I wouldn’t disagree if you look at these as a form of waste. But we don’t make them go away by avoiding the beneficial opportunities. We should, as time allows, learn, deploy, and utilize AI for good.

This task is diffuse, and largely about the participation of companies, and individuals in a very broad sense. It’s non-rivalrous with alignment research, and only mildly competitive with cyberdefense improvements. If a company cuts the security budget to fund a new product, that would suck. But most decisions aren’t like that, if the two budgets stay fixed, and the product budget is used more effectively, there’s no harm to cyberdefense.

Trusted Deployers

One component of this plan is formalizing an entity type, trusted deployers. I’ve previously discussed trusted deployers as a solution to the risks of open-weight models. The concept is useful more broadly. Closed models have deployment risks too. The implementation of security and guardrails still must be sound and sufficient. Evaluating models is an important step, but it could be argued that the most important step for regulation is evaluating deployment.

We should think of deployments as broadly as possible here. The obvious form of deployment, the public inference deployment, is but one type. I’ve identified four primary categories of deployment:

• Public Inference Deployments:

  • Definition: Model accessible by API to the general public. Should support both commercial (closed) offering and “freeware” offerings of open models.

  • Assume abuse intended. If a model has the capability of facilitating harm, denying that type of usage is critical.

  • Preserve broad accessibility. Denying value either universally, or limiting it to small groups would have harms. Highly capable open weight models cannot be made safe. But properly deployed “freeware” can be safe.

  • Competition and scale drive low prices. Deployers charge for compute usage and operational costs. Competition among deployers will achieve pricing as low as any other practical method. “Freeware” offerings come without any licensing markup.

  • Full safeguards. Hardware must be in a secure environment. Admin access must be tightly controlled. Identity sufficient to ban users attempting to circumvent protections.

• Limited Inference Deployments:

  • Definition: Model access limited to vetted groups. Deployments that insufficiently control access, cannot be classified as limited, and are treated as public deployments.

  • Enables intentional staging of availability. Cybersecurity programs like Daybreak and Glasswing improve defense before offense.

  • Enables internal evaluations. Models under evaluation have unique risks, such as disabled guardrails.

  • Requires rigorous Know Your Customer. Identity sufficient to vet users, and take action against for violations. Individuals and organizations must be trustworthy, and have a reason for access.

  • Scaling to use case and model capabilities: Vetting requirements dependent on the nature of deployment. Guardrails enabled or disabled? Which guardrails? Model capabilities?

• First Party Training Environments:

  • Definition: Environment for pre-training and reinforcement learning phases.

  • Focus on agentic training phases. Non-agentic phases must secure the weights. Agentic phases must constrain the agent.

  • Expect the unexpected. Each weight adjustment produces a novel untested model, even when following a well-trod path. Adversarial monitoring should complement reward functions. Special care when introducing new training methodologies, training larger models, or longer trajectories.

  • Checkpoints and Pauses. The unexpected should not be so expected that it goes unexamined. If adversarial monitoring discovers a warning sign in tool invocations, chain of thought, or activation monitoring, avoid training on these detections. Use pauses and checkpoints to minimize the active effect of monitoring.

  • Augment deployment standards. Standards for deployment should provide the base requirements. But neither is a subset of the other. Rather they have a shared ancestor. (Towards safety cases for frontier AI training | OpenAI)

  • Trustworthy independent evaluation is critical. This is the focal point of rivalries between model creators. Independent evaluation is critical as a check on these competitive pressures. Model creators will be under pressure to maintain secrecy here to avoid corporate espionage. There must be independent evaluators with internal standards sufficient to that level of trust. Trustworthy certification from the independent regulator with government oversight is one of the key contributions that no other party can provide.

  • National and global coordination focal point. This article discusses coordination, but does not offer a complete solution. Others have made useful suggestions. Ultimately though any solution must survive negotiation. I choose to leave that path for others. That said, the transparency enhancing effect of controls on training environments are likely to be a useful building block for any coordination solution.

• Third Party Training Offerings:

  • Definition: Model creators (first-party), provide customers (third-party) the services of custom training in various forms, mixing the first-party model with the third-party customizations.

  • Review third party inputs and trajectory. The third-party provides the inputs and determines the trajectory. This has to remain healthy though. First-party/trusted-deployer should provide guardrails to keep trajectory in line with safety parameters. Inputs should pass automated scans for backdoors or other harmful content.

  • Mostly not available at the frontier today. As of today, frontier-model third party training is not a current offering. We should anticipate future changes. We also should anticipate that as the sub-frontier moves forward, where third party training is more common, this category will become critical.

  • Secure portability. Third party customized models may need to avoid model clearinghouses. Trusted deployers should support secure mechanisms to transfer customized models between trusted deployers.

  • Most sensitive category: Deployments bring risk because a wider range of users brings more risk for malicious intent. Maintaining guardrails during training is a greater challenge than during deployment. In first-party training the single heavily invested party reduces the probability of malicious intent. A larger set of parties, with lower stakes, brings back some of that risk.

The deployment category is important for setting requirements. It must be combined with the capability level of the deployed model or model being trained.

AI Deployment Safety Levels (AIDSL)

I propose that a rating system, AI Deployment Safety Level (AIDSL), be developed. Each trusted deployer would be rated independently. An individual trusted deployer might offer different deployment services that would be rated independently.

When an AI deployment occurs, it has a sensitivity that is a function of the type of deployment and the model’s safety rating. Only when a deployment service has an AIDSL sufficient for that deployment type + model safety rating, should deployment proceed. For example a public deployment of a highly capable model, it should only proceed on a service with an AIDSL deemed acceptable to handle that type of deployment.

This model is based roughly on Biosafety Levels. Model ratings would likely be similar to the ASL rating Anthropic once used. Dissimilar to Anthropic’s ASL rating, this would not focus solely on frontier models. ASL as a rating failed because it attempted to anticipate future models, which was difficult. Decisions were made later as judgement calls, undermining the purpose of the rating.

When ratings are based on thresholds that the frontier has passed, evaluations are more stable mechanisms to assign ratings without allowing judgement calls. Frontier models would be evaluated under the capability threshold model rather than a forward guessing rating system.

Role of Independent Evaluators and Model Creators

Operating a deployment is a shared responsibility. Trusted deployers provide the foundation. Independent evaluators validate that foundation meets standards set by the industry and regulators. Model creators control access (within the limits of the deployment), contribute new models, revoke old ones. They may set model use fees, or manage subscriptions.

The model creators have even more to do in training environments, so evaluators likely have to evaluate both the environment and the model creators’ usage of it.

Model creators are likely to sometimes act as trusted deployers themselves. The total set of requirements does not change. They do not escape the need to operate safely when training internally or deploying internally.

The relationship between closed lab model creators and cloud providers already exists today. Today, the model creators evaluate cloud providers they either use for capacity or allow customers to use to access their models. What they evaluate though, and how they use the environment is not standardized. Additionally, only cloud providers review their deployments of open-weight models.

Many cloud providers participate in voluntary evaluations, like SOC 2 or FedRAMP. Customers then review these evaluations. Standards for model deployment should build upon or reference many of those existing computing compliance standards that deliver security and reliability today.

But this is a choice of each customer, not a broad protection against misuse. That gap in transparency needs to be closed, and the introduction of independent evaluators helps do that (OpenAI has outlined principles for effective third-party assessments).

Concentration of power concerns

Trusted deployers are likely to compete with each other. Competition without formalization already exists. As an extra measure, we could apply utility type rules, such as equality of access, like is done for electricity and internet data networks. This would probably happen if there was a dominant player, but it’d also be practical to apply it to a small group that’s dominant.

What does the open-model community lose?

The ideals of the open-model community are important. Some loss is unavoidable as those ideals conflict with safety. As a practical matter though, this proposal balances access with safety. Only the models that need protection are covered. Research organizations can make use of limited inference deployments and third party training, after KYC differentiates between trustable organizations and malicious users.

Privacy, in the sense that all monitoring is for a specific purpose, can be maintained. Yes, that privacy now requires some trust of the trusted deployer and the evaluator model. But ask where trust is more deserved? Is it more deserved by a system with intentional checks and balances, or amongst anonymous organizations that could be pursuing any purpose?

Does a trusted deployer need to be a compute provider?

No. They could be deploying to a cloud provider. If the combination of their deployment methodology and the cloud provider’s environment is verifiably secure, they might be virtual. That said, it’s a good expectation that most would be the actual compute provider. And it might be all, as there might be no companies interested in being virtualized trusted deployers.

This also extends to self-hosting labs. They might both own the full-stack, compute and deployment environment. They might also own a deployment environment that sits atop cloud compute. The key is the whole deployment must be evaluated. A deployment plan that assumes invalidated aspects of a compute environment is not a complete plan.

Agent Deployment & The “Passthrough Proxy” Problem

Today, agents are running in many places. Some are local, some in managed remote environments. Some in containers, some in VMs, some in user space. This is true, even when the agent utilizes a frontier model. A clever harness advancement, attached to a capable model, can amplify the model’s effective agentic capabilities yet further.

It thus, at some point, might become a risk to allow agents using the most capable models to be run in less secure places. This is above and beyond the user motivation toward isolation that promotes user safety.

Initially it seems like restricting external agents isn’t possible. If you can create any agent, you could use that agent to make simple queries and use it as a passthrough to enable an external agent. Considering this is wise, but there are responses.

When agents are deployed in safe locations, they can be monitored. An external agent using a safely deployed agent as a passthrough would have behavior patterns that would be recognizable. You wouldn’t be able to deny the first request from such an agent, but it should be possible to detect the patterns of successive turns. That detection doesn’t need to be perfect to undermine the practicality of an external agent.

You might ask, if you’re detecting multi-turn patterns, why not just look for malicious patterns. You can, and should, but agent-like patterns will be clear earlier than malicious agent patterns. When the risk is high enough, an earlier detection denies more capability.

There would be some resistance to that model, as some users prefer local agents for higher privacy. But like trusted deployers, external agents could still use lower risk models, which would still be very capable. We should also note that local agent deployment has risks, and the trend today is toward more and more agents being deployed in managed environments.

Capability Thresholds

We only want to restrict what’s necessary. Deploying models for robotics, and for local inference have purpose and value. Risk comes with the most advanced models. What brings risk may also change over time. Models that are too capable at first, may, through resilience, defense, or better understanding, become unnecessary to restrict.

I’d suggest an efficient method for measuring capabilities would adopt tiers to use hard won knowledge where we have it, and assumes uncertainty where we’re still learning.

Fully Open Tier: A base tier for generally safe models that are easily demonstrable.

• Restrictions: Limited restrictions. Freely downloadable weights. Liability for misuse would apply to end users or suppliers of products these are integrated into.

• Evidence: Utilizes existing evaluations. The necessary evaluations already exist. New evaluations will continue to appear as a side-effect of frontier development. To evaluate this tier, you need only catalog and curate these existing evaluations. You might optimize some, but none of this would be novel research.

• Cost: Realistically, there’s almost no burden introduced from such a requirement. Interest in the evaluations to establish capabilities already exists.

• Justification: A requirement for open-tier models to show they belong in the open-tier by being below a capability level, and above a safety level that these evaluations demonstrate, is a simple, cheap method to avoid more capable models trying to avoid requirements that should apply to them.

Open Deployment Tier: A middle tier for models that are similar to models with proven safety records, but require some controls.

• Restrictions: Must be deployed via Trusted Deployers with guardrails. Trusted Deployers liable for malicious use by default. With limited access deployments and proof of ability to pay for damages, customers could assume liability.

• Evidence: To avoid being classified in the frontier tier, a model’s capabilities should be under a capability threshold in comparison to the most advanced model by some margin. This margin will require regular re-evaluation by regulators. But for example, we might say 20% or 6 months. These evaluation requirements take effect when a model surpasses the fully open tier capability threshold or fails to pass a safety evaluation. Models should also be under a compute threshold. For example, no more than 90% of the total compute of the lowest compute investment of any model above a capability threshold. Compute thresholds are the primary means of assessing risk during training, before evaluation data is available.

• Cost: Deployment reduces flexibility on cost structures. But, with a well-functioning market, costs should be competitive to self hosted costs. Primary impact is blocking local inference as a potential deployment model.

• Justification: Too dangerous for open-weights, but safe enough to not need stronger restrictions and evaluations.

Frontier Tier: A top tier for models that do, or might, introduce new capabilities.

• Restrictions: Must be deployed via Trusted Deployers with guardrails at the highest AIDSL level. Trusted Deployers liable for malicious use by default. Model creators must share liability. Might be restricted to limited access deployments. Assume they will be restricted until fully evaluated. Training a frontier model should apply continuous monitoring. Transparency is required by including independent evaluators.

• Evidence: With frontier models evidence has to be treated with suspicion. Unexpected advancements could occur that existing evaluations might fail to capture. Safety models could have been incomplete. Evaluations must be continuous, and transparent. Existing evaluations are used as evidence to show a model must be evaluated as a frontier model. They cannot be assumed to demonstrate frontier models are safe. Frontier models should have frontier evaluations. New evaluations should proactively be created. Evaluations should actively look for dangerous capabilities. Continuous innovation in adversarial safety evaluations should be expected. Frontier models should respect Compute Thresholds, only surpassing existing records by pre-approved margins.

• Cost: Costs are significant if looked at in isolation. A complete accounting includes safety, which these costs contribute to.

• Justification: Frontier models are where the greatest risks emerge. Risks we might not know how to manage, or haven’t formalized the management process. The costs from frontier tier controls should be clearly justified. While pacing is a distinct topic, and we believe all of these requirements are justified independently of pacing, any slowdown they introduce should be more acceptable in that context.

At the lower tier, we want to allow free distribution of models without safety issues. Reusing the evaluation framework used to market and perform quality comparisons on the prior frontier provides a ready made, inexpensive way to define a threshold.

The open deployment tier probably would not target much today. Maybe GLM-5.3. But we should expect the need for that tier to increase as the current frontier is left behind and replaced by a new one.

The frontier tier is where challenges emerge, especially during training and internal usage. This is one of the major roles of the independent evaluator. This is not an entirely new role. Evaluations have been desirable and ever present in the release plans for every major model. It becomes more formal, consistent and non-optional. Consistency and formality should not mean static. They provide the base that proactive evolution occurs. Controls must keep pace with model capabilities.

Why Compute Thresholds Are Only a Secondary Layer

Early AI policy fixated on compute thresholds. Compute thresholds have two significant flaws. First, setting them without understanding the models themselves almost ensures they’ll be inaccurately targeted; too high or too low. Second, they could be bypassed by algorithmic improvements. Today, the probability of a single large jump in compute per training run is unlikely. The pace is partially fixed by the realities of compute expansion. Increases in training sizes will keep pushing capabilities forward, but if you see an unexpected leap, it’s more likely to come from an algorithmic improvement.

The exception here is when evaluation data isn’t available. This occurs when planning a new training. At this point, the effect of algorithmic improvements isn’t known and no evaluation data has been gathered. So compute thresholds, based upon prior training, becomes the primary threshold until other data is available.

Another case is global coordination, where we may find it difficult to trust and verify evaluations. We’d still want China performing evaluations, but when we go to verify compliance, it’s likely we’ll have more success at gathering the compute threshold data in a verifiable form. Ideally you want as many verification mechanisms as possible there.

Algorithmic and Recursive Improvement

The influence of algorithmic improvement grows if recursive self-improvement (RSI) becomes possible. While RSI could come via designing better hardware, or general scientific progress that enables higher quantities of hardware, those both would be very visible, and have real-world lags. Algorithmic improvement if unmonitored wouldn’t be as obvious. Also, we start to lose confidence in evaluations keeping pace if RSI is driving the pace.

One of the risks of RSI is that the decision point might not relate to any imminent catastrophic harm. The immediate event may be losing control of the process of improvement. Thus it’s obvious you should watch for that. But you can’t just watch, you must also react quickly enough.

Both are potentially harder than they sound. The monitoring may be hard to interpret. Rather than an absolute conclusion, it may arise as a risk. Even after you lose control, you may not know. There’s no necessary sign unless you attempt to stop the improvement process.

Once you start to lose control, the process of regaining control would become more difficult the longer improvement continues. What initially is a simple decision, escalates to more and more costly choices.

Another dimension is that it’s not enough to simply “not do” RSI. If a model has a capability to do RSI, you have to have a mechanism to prevent RSI from being done. If at some point you publish an open-weight model capable of RSI, you should expect someone to do it.

The conclusion here is that monitoring for RSI capability must be one type of evaluation. Models capable of RSI inherently are not safe for open release, whether or not RSI was utilized to create them.

Deriving and Applying Thresholds

You’ll note that I haven’t specified any specific threshold, but relationships. The relationships benefit from the knowledge we’ve gained so far. But I expect the evaluator and regulators to continuously consider if specific thresholds are needed, even if only temporarily.

So far, the negative impacts from model development have been minor or manageable. Because that’s true, any limit I proposed today, would be easy to suggest as premature. We can’t expect that to continue indefinitely.

When we do set specific thresholds, we should expect to be able to move them forward, when greater resilience allows. This is largely the plan for cybersecurity. If systems overall become more resilient, both at the most critical points, and in the broader sense, the risk level of models with moderate cybersecurity capabilities would decrease. There will be a sliding window here. How much we use that window is mostly up to us.

Alignment Safety

Alignment research is useful to users, but it’s critical to safety. Alignment is difficult, but so is monitoring for alignment. Monitoring for alignment becomes more difficult as capabilities increase. More complex models can misbehave in more complex ways, which are harder to detect.

Left to its own motivations, demand would push beyond safety. Monitoring and thresholds could catch this. But if you keep bumping up against them, they could fail. One failure could be a big problem if dangerous capabilities are latent. Ideally, you’d avoid bumping up against them, but instead, be ahead by design.

Resources for Safety Research

Left to their own devices, labs have devoted 5% of their budgets to alignment research. This number should be larger. Rather than specifying a fixed budget, my suggestion is that a larger ratio is mandated. Labs have a motivation to spend on capabilities research. When safety budgets automatically grow as a ratio, they simplify critical teams’ ability to access needed resources.

My starting suggestion is streams for alignment research, cyber research, and independent evaluators. It’s important to note that cyber research is not operational cyber. Operational security is integrated into each budget. Cybersecurity research should be for things like making the world more resilient to cyberattacks.

As an opening, I’ll suggest 30% be devoted to alignment safety research, 10% to cybersecurity research, and 10% to independent evaluators. I realize regulators, labs, and lawmakers will probably debate these, and they may be revised over time.

I’d also suggest that two things are tracked here, a monetary budget, and a compute budget. This helps avoid excessively large salaries or accounting tricks that do not focus on actual evaluations. The starting ratio should be drawn from current practice and updated periodically. My guess is that the ratio is something like 90% of costs are in compute.

A question is how to allocate the evaluator budget to actual evaluators. Early on, you might expect labs to create agreements with specific evaluators, as they are already doing. As time progresses you want to remove the leverage labs have over evaluators that this creates. The regulator body can certify the evaluators. They could assign specific evaluators at random or by some other mechanism to labs and trusted deployers. Labs could pay for the compute budget directly, assigning quotas to each evaluator, but otherwise not controlling usage. Labs could provide the monetary budget to the regulator to fund grants to the evaluators, splitting that between standing budgets, and assignment specific budgets.

Pacing Effects of Resource Allocation

One of the side-effects of allocating resources to safety is slowing the pace of capabilities research. Even with explosive growth, monetary budgets and compute budgets have some limits. So in addition to the outcomes of more safety research, you also place some limits on capability growth.

The ratios should adjust depending on how safe we’re feeling. Inside the AI Futures’ proposal, How to pace the US frontier, is what would be a 5:1 ratio for safety to capabilities research. If you sum my proposal, it’s only 1:1.

They include inference, which I’ve omitted. Including inference in the ratio fits better to international pacing agreements like they are attempting to solve. I believe their ratio is a better fit to the timeline they are targeting, whereas my timeline is assuming present day. Since I am assuming the international coordination is a later step, I’ve left inference out and started with a less aggressive ratio that might have immediate voluntary adoption.

Another source backing this structure, of using reallocation is IFP, in Should we “pace” AI self-improvement?.

We propose “pacing” — if it’s ever required — should consist of two parts:

1. Specifying which automated AI R&D activities are likely to pose severe risks, with thresholds carefully set based on rigorous analysis.

Because some disagreements about whether to pace AI R&D stem from different predictions about what level of AI R&D automation (and resulting acceleration) is even possible, identifying concrete thresholds might allow for different camps to reach positive-sum compromises.

2. Incentivizing the reallocation of resources away from those severely risky activities and towards two ends:

   1. Accelerating the diffusion of AI capabilities (including via developing applied AI tools like AlphaFold).

   2. Accelerating research that would make further AI R&D automation safer, either by:

      1. Improving the safety of AI models directly (e.g., via developing AI control protocols), or

      2. Boosting societal resilience to make the negative consequences of new AI capabilities less acute (e.g., via using AI to patch open-source code vulnerabilities).

Conclusion

There is a lot that can and should be done to improve safety and continue to use AI for good. But there is no single party responsible for all this work. This presents challenges and opportunities. As an opportunity, there’s many actors that can act independently on their own.

My contribution here is significantly around the concept of Trusted Deployers, which I place in the context of many other important actions. That concept, I believe, has the potential to resolve a rift that has developed from the narrative that either you have open-weights models or the closed models dominate. This is a false belief that these are either-or.

It’s not a costless decision, but the costs are much smaller than that either-or narrative creates. With alignment and cybersecurity, these are not trade-offs, but complements. The needs for resources they have are better met by demanding more. Rather than compete for a small pool, demand more from the larger capabilities and inference budgets. All kinds of safety are non-optional. The different timelines can make communication about “priorities” more complex, and accentuate a minor difference into a significant one, which is unnecessary.

Finally, while we each should be motivated to press our government to coordinate, to build functional capacity, and to work toward standards, this is not incompatible with making use of AI for good.

Sources:

1. Commerce Department Proposes Know Your Customer Requirements; Akin Gump; January 29th, 2024

2. An update on independent evaluators; Anthropic; October 15th, 2026

3. Designing a FINRA for Frontier AI; Mark Thomas; Lawfare; July 30th, 2026

4. Verbalizable Representations Form a Global Workspace in Language Models; Anthropic; July 2026

5. Uncensor any LLM with abliteration; Maxime Labonne; June 12th, 2024

6. The Great Refactor; Herbie Bradley and Girish Sastry; IFP; August 11th, 2025

7. How Should the US Prepare for Increasingly Automated AI R&D?; IFP; 2026

8. The AI-as-Normal-Technology view of loss-of-control incidents; Sayash Kapoor and Arvind Narayanan; September 14th, 2026

9. AI as Normal Technology; Arvind Narayanan and Sayash Kapoor; April 15th, 2025

10. Bending the Curve of Discovery: AI in Science Today and Tomorrow; Alex Imas, James Manyika; DeepMind Institute; October 8th, 2026

11. Introducing the Anthropic Cyber Mission; Anthropic; October 8th, 2026

12. Project Glasswing; Anthropic; April 7th, 2026

13. Daybreak; OpenAI; May 11th, 2026

14. Detecting misbehavior in frontier reasoning models; OpenAI; March 10th, 2025

15. Towards safety cases for frontier AI training; OpenAI; September 28th, 2026

16. Model backdoors are easy to create; Equistamp; Connor Tabarrok, Daniel Kang; October 9th, 2026

17. Anthropic’s Responsible Scaling Policy v3; Anthropic; February 24th, 2026

18. Priorities and principles for effective third party assessments; OpenAI; Lama Ahmad; September 22nd, 2026

19. Disrupting the first reported AI-orchestrated cyber espionage campaign; Anthropic; November 13th, 2025

20. How to pace the US frontier; AI Futures Project; August 2026

21. Should we “pace” AI self-improvement? Tim Fist and Saif Khan of IFP; August 9th, 2026

Related Articles

In May, I wrote about three challenges to preventing AI misuse. The list was not exhaustive. Since then, models have advanced. Anthropic demonstrates the risk from the recent GLM-5.3 release. The underlying challenges remain.

This update adds a remedy that minimizes the impacts to users, while addressing the core challenge of keeping dangerous models out of reach of those who would misuse them. Adopting this is urgent.

For a broader and deeper look at Cybersecurity and AI, I’ll suggest Why It Hasn’t Happened Yet.

The advancement of AI models has brought the capability of misuse. The potential for one type of misuse, offensive cyber activity, was demonstrated in dramatic fashion by the OpenAI HuggingFace incident. A lot should be done as I argue in Enough Reason to Act and Every Reward Bends. Alongside those efforts, three challenges need a response. It’s important that these are understood, as some solutions that initially seem reasonable, fail when exposed to these challenges.

Jurisdictions

The first challenge is jurisdictions that are beyond the reach of our law. The world has rogue states, lawless states, and aggressor states. These either turn a blind-eye toward harmful activity, lack the capability to enforce laws, or actively create targeted harm themselves. Existing laws cannot reliably reach actors that hide in these jurisdictions. There is a justified effort to close those gaps. There is slow progress. Sometimes gaps reopen. Because it’s a long running effort, we shouldn’t expect a near-term resolution, and treat it as a reality we must mitigate.

If our law cannot reach these places, we can try to prevent our tools from reaching them. Two further challenges, open-weight models and privacy hinder that effort.

Open-Weight Models

The key component to operating an AI model are the “weights”. These are the product of the expensive training process. The depth and quality of these weights determine how capable a model is. Currently many models are released as “open-weight”. In current practice, this means those weights are packaged as a file and posted publicly where anyone can download a copy.

Once a user has a copy of the weights, the ability to restrict or monitor them is very limited. They do require some computing resources to operate, and for the highest capability models this hardware is expensive. Operating at scale requires a lot of hardware and there are restrictions on where such hardware is shipped. But as substantial as that is at scale, the scale necessary to use models for harm is not that substantial. It’s reasonably within the reach of much smaller actors with very bad goals.

What is outside their reach, is the amount of compute required to create the weights. The training process requires levels of computation that are only available within two domains.

The first domain could be thought of as the US aligned domain. Top-tier compute is in the US, but production and second-tier compute extends to the EU, South Korea, Japan, and Taiwan. This domain is within the scope of practical US jurisdictional control. It’s a bit complex, and involves economic dependency, diplomatic alignment, but effectively the US has been able to exercise control.

China is the second domain. The US occupies the top tier, but China’s second tier is still sufficient to train dangerously capable models, especially if we extend that window out into the future.

Outside the US and Chinese domains, nowhere has the capability of training dangerously capable models. This collective grouping thus has the opportunity to control who has access to dangerously capable model-weights. This is well understood. I don’t ever see disagreement with this conceptually.

Where disagreement does arise is about open-weight models. The first reason for disagreement comes from not recognizing the danger. I outline the technical background of the danger in Security Can’t Wait and Deployments Can’t Wait.

The second reason for disagreement is misunderstanding the remedy. The typical assumption is that a remedy to the risks of open-weights would reduce the number of models available. But the appropriate remedy, trusted deployers, would not have that effect. Most users of open-weight models already use them in a way that would not be impacted. Others can easily adjust.

Trusted Deployers

Trusted deployers ensure guardrails are in place that reject requests that are harmful. Trusted deployers would host models from many providers. That would include those charging for access (closed models), and those released under an open license (open models). In this model, the open models would be distributed either directly to the trusted deployers, or via a clearinghouse (aka HuggingFace).

To receive a copy, you’d first have to be certified as a trusted deployer. This would start by showing:

• You have safeguards to keep the weights secure

• You employ monitoring and guardrails to detect attempts at malicious use and refuse them

• You do not allow fine-tuning that removes or weakens misuse refusal training

• You use monitoring to detect patterns of usage that attempt to “jailbreak”

• You use monitoring to detect patterns of attempts to perform malicious use

• You utilize these insights to terminate accounts associated with that disallowed behavior

Only models below a dangerous capability level would be publicly downloadable. But all open models would be available at every trusted deployer with sufficient guardrails. We might even suggest that all trusted deployers, or alternatively the largest trusted deployers, have an obligation to host all models that pass security reviews. This would treat these largest deployers as a utility with an obligation to provide access without discrimination. In this model, you would expect the number of models available to be as diverse as today.

Trusted deployers’ guardrails would use automated inspection of incoming requests, rejecting those deemed harmful. Deployers would be responsible for operating those guardrails. To support a variety of models, they would have base guardrails. Deployers might build their own base guardrails. Some would adopt an open-source guardrail project. Such projects already exist, though their maturity would need to continue advancing to keep up with adversarial demands. Likely the formalization of this model would help spur that.

An objection is that those guardrails are not likely to be foolproof. They don’t need to be foolproof though, they need to be effective. Guardrails raise the cost for attackers. They limit scale. They enable countermeasures.

The shift to a trusted deployer model does not radically change most users’ access to models currently released as open-weights. Most users already access models through cloud providers. Amazon Bedrock, Microsoft Foundry and Google Model Garden, host third-party models, including leading open-weights models. Those providers will become trusted deployers.

In the trusted deployer model, model weight files are shared only with the trusted deployers. Users do not have access to the model weight files. But most users never access them today. Most users do not have the compute capacity to use those models, so they never touched those model files as-is.

The largest collections of compute are at cloud providers. Those would all be accessible by the trusted deployer model. Smaller groupings of compute – private data centers, colocation facilities, sovereign national infrastructure – would need to be adapted to a trusted deployment model. Most would do so.

Open-weight models would continue to exist. The trusted deployment model would only be a requirement for levels above dangerous capability levels. Most usage of highly capable models already goes through providers. The compute requirements leave few other options. The exceptions are powerful organizations that can become their own trusted deployers.

For this compromise, we’d earn the capability to ensure dangerous capabilities are never deployed to servers where we lack control. Dangerous models would not have their model-weight files posted publicly. This restriction would earn security, with minimal change to actual operations.

Privacy

A trusted deployer would be expected to do more than deploy passive countermeasures. They would also be expected to use identity as a tool to actively restrict the attempt at harmful use. If attackers are allowed to endlessly probe and retry defenses, their chance of success goes up. Using conventional identity systems can reduce access. When abuse patterns are detected, those accounts are deactivated.

The best model for this though has a strong identity. Weak identity systems allow fabrication of new identities. The default state of anonymity on the Internet has costs by creating weak identity. Privacy advocates attempt to maintain this state. I, like some others, believe the costs of this anonymity as a policy are too high. This isn’t specific to AI, but it does relate.

Like other protections, we shouldn’t expect foolproof systems. A stronger identity system would add costs to an attacker trying to manufacture identities. They could buy accounts from a real person. But the cost would be such that using these accounts to probe defenses would no longer be economical. They would become more cautious and less aggressive about how they used those accounts. Attempts to access dangerous capabilities would become rarer, allowing for more active response to the remaining attempts.

To enable strong identity, we would need to change some of our expectations about creating accounts with trusted deployers. Trusted deployers should be both trusted and expected to use automated means for scanning request intent. They should be both trusted and expected to know who their users are as individuals, or as organizations. Part of the certification process would ensure that each individual operator within the trusted deployer does not access this information. The exception being the review of requests already flagged as malicious.

We should be pragmatic, but we’ve been idealistic. When countermeasures can’t be implemented due to obscuring the lowest layers of a technical stack, we fail to achieve privacy and prevent harm. If service providers always knew who was using their service, they’d be able to deny access to anyone detected acting maliciously in the past. But the internet offers too much anonymity. Providers can shut down an account, but without accounts tied to a strong identity, a new one can be created. The current standard is too lax about this. We could make it more costly for attackers to maintain access.

China

As mentioned, China is the second domain where model training is likely. For the near future, and potentially longer, this would be second-tier, which provides some security. But that gap is not large enough to take for granted. Misuse will be far more preventable with China’s cooperation and adoption of a trusted deployment model, in contrast to the currently prevalent open-weight model.

While any agreement with China is not without challenges, this specific proposal has advantages to China. They also face risks of misuse and should be motivated to prevent it. It’s far less challenging than the global coordination needed for pacing AI development globally, a critical action to address loss-of-control risks.

Conclusion

Trusted deployers resolve many challenges to preventing AI misuse.

Jurisdictions, open-weight models and privacy complicate preventing AI misuse. Jurisdictions won’t change, model choice and privacy we’ll continue to value. These three challenges compound each other. Open-weight models place powerful tools in jurisdictions beyond legal reach, while anonymity makes it difficult to detect or deny access to bad actors even where laws do apply. Treating any of these in isolation understates the problem.

Yet safety must also be a priority. Trusted deployers enable broad and diverse model access while partially resolving the challenges. They are also functionally similar to prevailing usage patterns. This makes it an excellent form of compromise.

This does not make it an easy compromise though. Developing a trusted deployer model takes time, even though we already have examples of it in practice. It has to expand to become the standard across all jurisdictions with enough compute capacity to train models.

Meaningful identity verification will feel like a concession on privacy, because it is one. Trusted deployments won’t affect most users, but they will frustrate some. Coordinating across jurisdictions will be slow and incomplete. None of these are reasons to avoid acting, but they are reasons to be honest about what any given measure can and cannot achieve.

What’s unacceptable is the current default. Hard choices cannot be deferred. Privacy and safety will be at risk without appropriate measures. Treating anonymity as an unqualified good has costs. We should discuss, agree to, formalize, and adopt measures that strike the right balance. We should start that immediately, as each step takes time.

Related Articles

My interest is the reason for acting. I make the case for that in The Case for Action. But first, I address eight mistakes that are used as arguments against acting. Four of these are, Mistakes of Detail, false assumptions or mistaken facts. Four are Mistakes of Form, modes of reasoning that fail.

The core of the news cycle focuses on loss of control, aka rogue AI. One of the turning points was Dario Amodei, CEO of Anthropic, publishing, We Must Pace the Frontier. In that, Amodei mentions several risks:

the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption.

So far, I’ve covered cybersecurity (Every Reward Bends, Nobody Was Watching, Security Can’t Wait). Partly that’s because my expertise is stronger there. Partly because it requires broad action across information technology, and I can be more persuasive there. The many small incremental activities lend themselves more to the type of influence I have today.

This is my first serious effort at discussing loss of control. The most pressing aspect is that the important steps take time to arrange. They are difficult, and become more difficult if rushed. The direct actors are fewer and more centralized, but political support is necessary. That requires broad understanding that takes time to reach.

The timeline is less definitive. The next 12-18 months will be bumpy in cybersecurity. Loss of control, if it does emerge, might be farther away. But there is a chance it’s equally close. Warning signals can compel action, but they can also compel overreaction. To avoid that, we want to see broad awareness and creation of systems that facilitate coordination that mature. If we have time, they’ll mature smoothly, if we don’t we’ll be more prepared than otherwise.

A Title More of Us Could Agree With

The focal point of loss of control as a narrative is the book If Anyone Builds It, Everyone Dies, a 2025 book from Eliezer Yudkowsky & Nate Soares. It’s an eye-catching title. If I were publishing the same book, I might accept the same title, because, you know, titles. But unlike Yudkowsky and Soares, I do not accept the literal meaning.

I can iterate this one clause at a time toward something with both my full agreement and broader acceptance. My first iteration adds uncertainty:

If anyone builds it, then (there is a chance) everyone dies

I consider five possibilities:

(a) we successfully control superintelligence by intentional alignment
(b) we accidentally create a benevolent AI overlord
(c) we accidentally create a non-benevolent AI overlord
(d) we accidentally create an AI that destroys human civilization
(e) we do (d), (c) or (b), but before losing all physical control manage to enact some hail-mary or emergency plan that actually works at destroying (b), (c) or (d).

It’s tempting to try and assign probabilities to each of these. They would have large error bars if I did. If it were the only iteration, I’d do that. But the following iterations decrease those probabilities’ relevance.

If anyone builds it, then (there is a chance) everyone dies, (but if we build it right, that chance is less)

We have agency here. Those error bars will stay high, but we can push the probabilities down. You might still say, why take any risk, if we don’t build it, no risk.

If anyone builds it (and let’s be honest, someone is going to want to build it), then (there is a chance) everyone dies, (but if we build it right, that chance is less)

The world is a big place, with a lot of people. Even if you and I believed the strongest version, (d), not everyone would, nor would we be able to convince them all.

Why would anyone still build it? Well, because both the accidental benevolent overlord (b) and especially deliberate alignment (a) appeal. Some reject (b) on a value basis. In a strict interpretation, I wouldn’t. You might even argue a large part of humanity has hoped this is true, but in supernatural form. I would have a problem with those that object to (b) being forced into it though. But it’s pretty easy to imagine a group unconcerned with either of our views, and how can you fully object to (a)?

So, if there’s going to be a lot of people who want to try at (a) or (b), you could try and stop them. But again, the world is big, so you need very wide agreement for that to be practical. And when they do try, you’ll be stuck with whatever that attempt produces.

If anyone builds it (and let’s be honest, someone is going to want to build it), then (there is a chance) everyone dies, (but if we build it right, that chance is less) (so we should be part of building it, so we can do it right)

But if (a) is possible, or you can discover if it is possible, you can try and do that. And if you trust yourself more than some other segment of humanity, you can try and make (a) more likely, or prove it’s impossible, which could convince everyone to stop trying.

Is it hubristic to trust yourself? Yes, or at least it flirts with it. Is it avoidable? Only if you trust someone else to build it better.

If every modest person defers, the hubristic take the wheel. So, the capable but modest must embrace some self-trust and hope they haven’t misjudged themselves.

Seeking power for power’s sake is bad, and it’s easy to lose self-control in that process, but seeking power for an outcome is also a necessity. You can distrust this a little, but you can’t distrust it absolutely.

If anyone builds it (and let’s be honest, someone is going to want to build it), then (there is a chance) everyone dies, (but if we build it right, that chance is less) (so we should be part of building it, so we can do it right) (but we also need to not be under pressure to do it fast, or even our good intentions will fail)

When you look at those from labs calling for pacing the frontier, their views are closer to the iterated title. Soares and Yudkowsky may believe in the simple version, and you may disagree with that. But that wouldn’t give you a reason to disagree with the more qualified version. And people in the labs who agree with the qualified version are suggesting a slowdown as critical to finding a solution.

The control necessary to permanently block everyone is higher than adding extra time by going slower. Going slower gives more time to do it right. It gives more time to discover what’s possible. You may even discover that superintelligence is impossible, or much farther away. I find the first very unlikely, and the second is becoming less probable, but who knows?

We should take that full concept seriously. We still have to decide how much any individual believes this. We still have to decide which ones handle pressure the best. We still have to decide which ones are actually most competent at building something safe. And we still have to get the “someones” who aren’t part of this group to agree, or take away their ability to succeed (quickly).

I think we’re capable of coming to a solution here. But if we rush, out of complacency, or because we’re fighting over something else, we’re far more likely to fail.

Mistakes of Detail

Before I make my case for action, I want to address some common mistakes. These are details where it’s easy to misunderstand the general case for action. I’ve seen these often enough to think there’s a chance you might also make them. I do not know you will, but if by naming them I give you a better chance, it’s valuable.

Presume we all have to die

A common mistake is to say, I don’t see how AIs could kill us all. By the time they are half done the computers will go off and they’ll have to stop. Or something like that. This objection is irrelevant. We don’t want any large group of deaths.

I’m not against asking the question, because you can learn from it, and what you learn can be used in places you didn’t anticipate. If you ask the question, I’d point out (a) once civilization starts collapsing our chance of killing ourselves, always a risk, goes up, (b) you have not, and will not, think through every possibility. It shouldn’t be necessary to explain how a superintelligent system is going to escape from its physical limits after it escapes from our virtual limits.

There will be possibilities none of us can anticipate. Using the worst case to dismiss all the lesser but still bad cases is not an effective form of reasoning.

Presume false competition

Loss of control is one topic. But it’s not the only one. Dario mentioned four. One mistake I’ve seen is thinking there’s a competition between them. Some people try to spin that as a reason to ignore loss of control.

If you ignore probability and timelines, loss of control is the most significant. Cybersecurity is the most time sensitive and requires the broadest set of actors to respond. Bioterrorism requires DNA labs, a very special kind of actor, to respond, and good deployment and training practices by AI labs. Economic disruption hasn’t really emerged, and it’s the one part I disagree with Dario.

The actors who are most pressured to respond to all of these at once are the AI labs themselves. In many cases there are coordination challenges with incompressible durations, no matter how much effort is expended. Global coordination of pacing might accelerate with shared priority, but there’s no unilateral shortcut.

Cybersecurity does fall into a unique slot: unmitigated, its effects land in a year. Rogue AI has a broader timeline. We don’t know when or if capabilities increase to the level that makes it possible. It’s a good bet they can, but not in the next 12 months.

Presume cybersecurity and loss-of-control are the same

A pattern I’ve seen goes like this: Cybersecurity risks could be managed by diligence. I can understand how those risks materialize. I think I can apply that to loss-of-control.

It’s great you’re thinking of cybersecurity. It’s important. Cybersecurity needs lots of small incremental changes, starting immediately. We need a unified effort and commitment to doing that. We need great tools to make it efficient. If we do nothing, we could end up with catastrophes.

But the loss-of-control risk takes place in a different world where things have changed. We don’t understand that world yet. It might be that this different world is harder to enter, and even if we try it could be a long way away. But I see no reason to be confident of that. We’re close enough that we’re learning some things. Cybersecurity applies to this world, but it’s also going to be radically changed.

So no, cybersecurity isn’t enough as a solution. I think it’s incredibly important. I think we can and will respond. If we didn’t we’d have chaos that would make every other bad outcome a bit more likely. If we really focus over the next 12-18 months, I expect we’ll enter a more stable period that’s more secure and better managed.

But it’s not one challenge. Being able to handle the cybersecurity challenges of the next 12-18 months is not the same as being able to handle loss-of-control. Good cybersecurity could avoid a few mistakes here. But it’s not a full response and can’t substitute for one. Partly that’s because handling cybersecurity is going to depend on having access to more capable models than the attackers. If the model is the attacker, and it’s the most capable model, that advantage is gone.

Those guys are weird

I have to thank Matthew Yglesias for highlighting this trend, in A simple plan to save the world from rogue AI. Once mentioned, it’s impossible not to see. There are eccentric people, and eccentric opinions. Being an eccentric person doesn’t make your opinion invalid. A lot of uninformed dismissals take the form of attacks on Effective Altruists, often bringing up Sam Bankman-Fried. Yudkowsky has strong opinions and does eccentric things. The concept of being analytical about altruism, vs. doing what looks good / feels good, doesn’t seem like it should be eccentric, but through history it turns out it is. And yes, if you drop into San Francisco, you’ll find some eccentric lifestyle choices. But for San Francisco, if I know the history, that’s actually more typical than atypical. Either way, none of it is a successful argument about risks.

More importantly, the existence of an eccentric opinion doesn’t make the milder, more defensible version of that wrong. The milder, more defensible version of loss-of-control is still worth paying attention to. 10% chance of end-game is pretty important. Strange as it is, it can require balancing, but dismissing it entirely would be a mistake.

Mistakes of Form

The next set of mistakes are about the form of reasoning. They are more complex as a result. What’s ultimately most concerning about these, is that they are used to cut short a search for understanding. The first four fit a learning experience, but the next four, cut that short, substituting a dismissal. If accepted they lead to a stark choice, of no action, or an overreaction as the only remaining choice.

Some are patterns that reappear enough to be well-known. This is not, however, an exhaustive list. I call out these mistakes specifically because I’ve seen them used to shut down best-effort conversations.

Reasoning from cynicism

Reasoning from cynicism is not a sound mode of reasoning. As a tool to identify concerns, it’s perfectly functional. But it’s not a complete form of reasoning. For any given situation, if you use cynicism as a form of reasoning, you’ll arrive at different conclusions based on which actor you target your cynicism at. And many of those conclusions will be directly contradictory. The truth may not be represented by any of them.

I’ll hold out Matt Stoller as an example, telling us to Stop Panicking About AI. This isn’t a generalized attack on Matt. He serves as an example because he made mistakes, used cynicism as a component of his reasoning, and got a lot of attention.

He knows he’s not an expert on this topic, yet is willing to make fact-free proclamations like “Agentic AI is dangerous, but we can improve the products to limit damage.“ How does he know that? His fix has no clear line back to safety.

I have no problem with Matt adding his view about copyrights. Dismissing the views of the better informed is unnecessary for that. But the post has over 600 likes. Is it his pre-existing audience? Is it creators defending their turf?

Those are the hopeful answers, because the post is not a good discussion of AI. He competently recounts the events that are available elsewhere, but the analysis doesn’t engage with the actual topic. As a response it’s slop, latching onto a live story to re-promote an existing point of view.

One of the ironies is that his prescribed fix is for us to abandon a “nihilistic refusal to govern for the public interest”. I’d be happy to. The irony is that this nihilism is closely related to the cynicism we (the public) have applied to politics. I sit in an uncomfortable middle, thinking mass-cynicism of the concept of government is a mistake, but sharing the same feeling about mass-cynicism of tech companies. You have to be willing to acknowledge the possibility of a good politician or a good company for there to be any room for them to succeed.

Reasoning from markets

Markets are useful tools, but at best they process information, not create it. So when predictions about impacts from AI are made, it’d be a mistake to use a lack of market signals to dismiss them.

I read Marginal Revolution regularly. I’ve had disagreements with Tyler Cowen or Alex Tabarrok, but generally found the starting argument informative. But, recently Tyler has been suggesting AI and technology experts are making predictions unscientifically.

In “Act like it is science” he asks:

1. Your outline of how, say two or three years from now, we might estimate the additional cybersecurity costs from the AI break-ins. I am convinced that number is not zero, but give me your method please. If it helps, here is an estimate of past cybersecurity costs, done by top economists.

2. Your current numerical estimate of what those costs might end up being, of course to be tested against what actually happens over time. Obviously, you can do this for a few different regulatory/safety scenarios. This is one simple way to prove yourself largely correct, albeit with a lag. (NB: you do not have to take this as a substitute for your preferred safety measures. But please try to be specific in your predictions.)

3. A list of what stocks or other assets you have shorted, now. Obviously if your answer to #2 is sufficiently low, you could answer here zero, as I would do. I expect costs, but not so high that we cannot muddle through and have expected positive stock returns.

He’s since been doubling, trebling and quadrupling down, posting cybersecurity stock price charts and saying that with them “we are getting somewhere concrete and scientific rather than just scare stories.”

(1) sounds reasonable. But Tyler is responding to events that stirred both loss of control and cybersecurity concerns, and conflating the two.

Applying that request to the Soares and Yudkowsky version, that everyone dies, is absurd. The cost of everyone dying, crudely, is 8 billion x $14M = $112 quadrillion. That number is useless for any argument about what to do next. Historical data does not provide a probability, and we would be surprised to ever accumulate enough data to retroactively apply.

This problem of having no historical precedent doesn’t go away. Instead of one, all encompassing number, you start having many scenarios, each with an outcome (e.g. 1% die, so only $1 quadrillion). But each is just as useless. The concept is more useful than the number.

The second problem emerges in (2) and (3). We’re only ever going to get one version of “what actually happens over time”. But anyone who takes (1) seriously isn’t going to wait around. They are trying to change what actually happens. And (3) is a trust test, indicating Tyler is not going to take answers to (1) seriously, unless you put your money where your mouth is.

The preparedness paradox disconnects (1) and (3). Tyler can’t be fully unaware of this as in (2) he allows “do this for a few different regulatory/safety scenarios”. But you can’t pull those multiple scenarios forward to (3) which demands “your answer to #2” as a single number, without locking in probabilities that people will/won’t act.

If you follow his posts after, it’s clear that he’s not avoiding that trap, and (1) is just an opening salvo to draw you using insurance rates and stock prices instead of stories about events without precedent. Probably this is unintentional, and just derives from a habit of reasoning from markets in unfamiliar fields.

I do see a trend in Tyler’s posts toward searching for evidence, rather than dismissing evidence. But if he wants to do so, he needs to disclaim the original rather than just quietly walk away from it. This would help avoid others being drawn into the trap. He must also be open to non-market based predictions.

Shorts don’t tell you someone’s commitments to action

Shorts are an unfair test of credibility. Think through your options after you spot a risk:

A pure trade. You short, wait for the disaster, and collect mega money. You don’t need to do anything to divert. At best, you send a cryptic signal. If someone decodes the signal, they can trade on it too without taking action.

A warning without trades. You explain the risk, persuade others, plans are made and action is taken. You send a clear signal, no decoding needed. Action is the intended outcome. The beneficiaries are those who would have been hurt.

These are the simple options, but we can combine them:

A warning monetized. You short, then explain the risk and persuade others. Those that believe the risk won’t be managed act by selling, causing the price to drop. You exit the short, taking profits. Plans are made, action is taken, the risk is fully mitigated, and prices return to normal. The people who would have been hurt still benefit from the warning, but now you do too. The sellers end up a bit worse off, though it’s worth considering who they are. If they held the stock until the disaster that’s realized in the pure trade, they probably would have been hurt in that way.

You’ll notice that in a warning without trades, I didn’t mention sellers. It’s not core to that story, but realistically, if they appear in a warning monetized they’d appear there too. But there’s a reason they might not appear in either:

A warning where trades don’t pay. You short, then explain the risk and persuade others. Others expect plans to be made and action to be taken, and don’t sell, expecting those to result in opportunity loss. Prices don’t change. Plans are made, actions are taken. Your short fails as there was no disaster, and no opportunity to exit the position.

This scenario is bad for you, and once you’ve disclosed, there’s little you can do about it, other than manipulating views on the probability of action. In reality, it’s never quite so dire for short sellers, as there’s always a mix of opinions. But timing the exit is important.

A warning where opportunity is missed. You short, then explain the risk and persuade others. Those that believe the risk won’t be managed act by selling, causing the price to drop. You don’t exit the short. Plans are made, action is taken, the risk is fully mitigated, and prices return to normal. Your short fails as you missed the chance to exit.

If the risk is unavoidable or only partially mitigatable, there is always some residual payoff. But if we’re talking about risks that disrupt society, the silver lining may be inadequate. You may experience just as bad effects from the spillover as any residual payoff. And, well, everyone else suffers, so if you care about that, you really want plans that fully mitigate.

If the risk has low spillover, your best bet in shorting is the pure trade. If it has modest or more spillover, your best bet is a warning monetized, exiting prices reflect less confidence in mitigation than you have. If however, you expect prices will never dip to that point, you should never short in the first place.

It’s a little more complicated, because as discussion about a risk occurs, you learn more about the probability of mitigation. But this in-between period isn’t important to the question of whether you really can reason from markets. The only thing you’re guaranteed to learn is how persuaded people are and how that affects their participation in mitigation. There’s no guarantee you’ll learn anything new about the risk itself. If you do, it will be in the form of words, not prices. It will come from a person who has reasoned through the scenario. That person’s views could end up in prices, but you’d have no way to separate it from the other views that go into prices, except by listening to their reasoning.

You should not, in any direct way, be updating your views about the merits of arguments for risk, based on stock prices. At best, changes in stock prices are a signal of a change in opinion. Whose opinion and why? You don’t know. Better to just go to the source.

Insurance rates are also indirect

In another attempt Tyler Cowen suggests historical market prices and cybersecurity insurance rates are sufficient evidence that there’s not much to worry about. He assumes the market has already priced in the effects, that it’s also simultaneously going to force action.

The first problem is if insurance rates only reflect the views of insurance actuaries. If they aren’t yet convinced of a risk, rates won’t yet have changed. If you want them convinced, you either have to persuade them, or hope they persuade themselves.

The second problem is insurance actuaries also have opinions about the probabilities of mitigation. If you persuade them, you’ll likely persuade others. Rates would come back to normal the more they are convinced that mitigation is likely. Rates only reflect unmitigated risks and only to the degree that it’s likely to remain unmitigated up until an actual event.

The first and second problems are both affected, but in opposite directions, by the capability level of the actuaries. If they are wise all-knowing oracles about risks, they’d be wise all-knowing oracles about mitigation. They’d also have a responsibility to explain their rate setting process, which would disclose the risks. That would, assuming they are mitigatable, lead to mitigating those risks. They’d only fail to lead to mitigation of a mitigable event if their disclosures weren’t trusted. That would be odd if you accepted them as all knowing oracles.

The third problem is assuming the structure of finance isn’t hiding the risk. Finance has lots of fine print, and the cul-de-sacs of limited liability can swallow a lot of risk if strategically placed. There is the counter-argument to this that if there was a weakness, someone would step in and offer a new type of insurance or finance instrument to span it. But that’s in theory. In practice we have evidence of how this can fail. And keep in mind, there is a reason this is my third, not first problem.

Markets accumulate insight, they don’t produce it or disseminate it

If you’re not asking the hard questions, place faith in markets over reason, you’re in a circular type of reasoning that is a lazy response. If no investor is willing to hear the stories, there’s no reason for prices to change. If you successfully persuade investors to ignore the story, because prices haven’t changed, they won’t change until it’s no longer a story.

A perfectly functional market might be described as a representation of collective reasoning. But not only are markets not perfectly functional, collective reasoning has to begin with individual reasoning. Relying on the market could be a lazy way to capture others’ reasoning. But any appropriate update has to start from an individual, and then spread by other independent reasoning, or by person to person making arguments that don’t reference a market. Lazy followers will eventually adopt the same view once non-followers update their views, but the followers’ information will be delayed.

Demanding yes-no answers

I wouldn’t claim I’ve proven anything here. Proving we’d lose control, or proving we’ll fail to respond to cybersecurity isn’t my intent. I don’t know that it’s even possible. These types of topics don’t adapt well to yes/no answers. It’s true in many domains, and very true here. Life is largely in the grey zone, and pretending like it isn’t only gets you so far.

We have a problem, with a public, that at an aggregate level–aggregating both individuals and individual actions–is far too partial toward yes/no explanations. It’s a tendency that has rewired our media, our politicians, even our personal interactions. It’s so pervasive, we take it for granted in most cases. When someone does bring it up, it’s treated as quixotic. Maybe I am tilting at windmills to make that argument.

It’s possible that the approach, or views, of Soares and Yudkowsky are more effective than mine in this world where yes/no thinking is as common as it is. I had to provide a long explanation, and a set of qualifiers. My book title would never have made it on the bestsellers list.

Another problem with my approach is I’ve just insulted the reasoning systems of a lot of people. That’s going to seem elitist if anything does. But I do it because I think they can change. I believe in them, and any resemblance to insulting a person is incidental. I don’t want to critique you, I only want to critique a reasoning system that ends in bad results.

The thing about “inexorable forces” is that many of them aren’t. They are powerful, hard to resist forces. But you can resist. If you want to successfully resist, you better plant your feet more than 2 inches away from the cliff.

For things that are difficult and are going to take time, you have to start earlier, and often work concurrently, taking care of other in the moment issues while making progress on the harder ones. It’d be nice to be able to focus on one issue at a time, but that’s just not how life goes. You may ignore and delay some big difficult things, but if you do, it’s primarily because they just aren’t important enough.

Importance here has a bit less to do with probability than other places. Early effort costs less than late effort. You commit to the big effort at a time when probability is more certain. The cost of trying to make up for missing the early effort, goes up with delays. So if it might be important in the future, you want to invest in things that shorten that future timeline.

We already have missed some opportunities. The most significant one is to have government institutions that understand this stuff. They would be delegated to create a plan, in collaboration, and when confident in that plan, move forward. Right now we don’t have the careful drafting of plans, we have reactions.

Letting forcing functions win

The non-benevolent, superintelligent, super powerful AI risk has two versions. One is the event itself, anchored to the moment control is lost. The other version is worldly pressures forcing us to handle that pivotal moment poorly.

There’s some version of this that will always be about doing things in the moment, while being in a fog where we aren’t entirely sure if a superintelligent AI exists, or how close it is. That version relies on momentary things like, monitoring deployments, securing sandboxes, running evaluations. These are the things that if there is a precipice and we approach its edge, we’d see the edge. Logic says we stop there and admire the view.

There is another version that is about doing things in the moment, knowing that logic could fail, and a crowd, wanting to look over the edge themselves, might push us off. That version is about creating verifiable compute, about discussing the edge with places like China, about having a lot more people who understand some of this. Those three, are actually in order of difficulty, from least to most.

There are two ways you can let forcing functions win. First, you can give up, and not fight. Second, you can ignore that they exist and end up dominated by them. You have to be aware they exist and willing to work against a deck that isn’t in your favor if you want to take the reins and choose a path of your own choosing.

The Case for Action

So far, I’ve only made the case for action obliquely, referring to others’ arguments and explaining how there’s a consistency between working in AI, and wanting it to be paced. On the cybersecurity side, I’ve done that already, in Why It Hasn’t Happened Yet.

On the loss of control side, I haven’t. One of the critiques you see applied to the Soares and Yudkowsky version is specifics about how AI would get out of control, start to harm us, stay out of control, and end up killing all of us. I could provide a story like that, but it would have a sci-fi tinge where you could poke holes at. It’s more useful to think about how lesser forms of those stories are very possible, and that the consequences before regaining control, and the costs to regain control, are quite problematic.

We could go back and forth debating these stories, and likely never have a full resolution. The defenders would have holes where they couldn’t promise defense, and the rogue AI story would have holes where some unexplained jump would be necessary, given current capabilities and what humans have currently wired AI into.

This sequence of stories has less to do with absolute answers than possibilities. Those possibilities then connect with trajectories. One trajectory is how future possibilities are emerging. A second trajectory is our preparations. And the third trajectory is how quickly we can change either of these trajectories.

The OpenAI HuggingFace event revealed important information about these trajectories. It showed that labs were being reckless. If that continues we’ll be less prepared for whatever comes. It showed we had no structure to reduce recklessness. If that continues, we’ll do less preparation.

We also have the issue that the general public remains mostly uninformed about how information technology, especially this technology, works. It’s been a blind spot in our culture that accurate depictions of software developers are missing from popular culture. With that understanding missing, the popular reaction to events will both over and under react, and fail to choose the best nuanced response.

The risks to cybersecurity were evident within the technology domain before the OpenAI HuggingFace event. But another change from that event was to provide a demonstration that did not require technology understanding. From this, the public became aware of capabilities they would otherwise have struggled to understand.

That’s a big hurdle to overcome, and it’s just the first one. Creating a US regulatory system would take time. Many people who don’t understand each other’s positions need to understand them. A lot of details that don’t have shared understanding need to be agreed to. And then that’s followed by an international system, which has an even wider gulf and has to somehow co-exist with a set of standing disagreements at that level.

All of this should advise starting now. That doesn’t demand an immediate overreaction. In some ways, an early start is a way to avoid overreactions later. More importantly, it makes underreaction less likely and less inevitable. With a late start, the choice of actions immediately available may be poor choices all around.

An event might happen where the only reasonable action is shutting things down. That might go as broad as huge numbers of servers, or shutting down parts of the internet. With such a stark choice, inaction might take hold and nothing gets done until things get worse.

How bad would it be if we did nothing?

I think we will take action. I speak about it to make that more likely. I believe others do too. I believe all this will mitigate many risks. So the risks I mention next, those aren’t risks I think will happen. They are risks I think we’ll avoid. If we do the bare minimum we will end up with some ugly costs.

Existing cybersecurity depends on successful layering of imperfect layers. Attackers have to search through every layer and compose attacks to perform their malicious activity. AI enables the search and composition.

The capabilities there are such that if nothing had been done, we would be seeing widespread successful cyberattacks. Fortunately, while more is needed, much has been done. The infrastructure for implementing cyber defense has history and maturity. AI was used to find, fix and deploy many critical vulnerabilities. Additionally, all the leading AI labs took responsible actions to restrict attackers’ attempts to use AI for composition of attacks.

For cybersecurity, you need to worry about how failures compound. For financially motivated attackers, if the success rate of an attack doubled, not only would you see twice as many successful attacks from existing efforts, but since the efficiency increased, you’ll have more effort. Second to that, the tax on revenue and then profits would accelerate as more are sapped by attackers. If you start with 1% of revenue, then double to 2% when success increases, then double again to 4% when attackers are incentivized to invest more. An average profit margin of 8% would be cut in half by those effects.

That kind of effect would force a reorganization of many business operations. That forced reorganization would create costs in any situation, but occurring suddenly, it might be poorly managed and result in systemic failures.

The loss of control risks could be described two ways. We could talk about the cost of effects a rogue AI might take. And then we can talk about the costs of the reactions we’d need to regain control. This assumes we do regain control.

I think people are generally right when they suggest that a rogue AI, at the current capability level, would only do limited damage. The costs to shut it down might be trivial, though there are scenarios where something worm-like emerges that requires wide scale computing outages to respond to. Generally though, concerns about rogue AI don’t focus on current capabilities. They worry about our ability to slow down or stop capabilities progress once we commit to it.

Without anything like global coordination, there are significant loose-ends to controlling capabilities progress. Some additional capabilities might emerge from low-compute evolutions, like better harnesses.

The actual list of bad things a rogue AI could do is quite long, in the same way as the list of bad things a person could do. What’s the minimum cost you’d need to suggest to merit an AI safety program that redirects half of all training capacity toward safety? Two thirds?

Plan A uses this mechanism, detailed in its appendix.

What are we really giving up?

Absolute outcomes like “everyone dies” can justify absolutely any response. The only logical limit to a response that avoids that is a less costly response that also avoids the outcome. But I’ve conceded that the outcomes may be less than absolute, that those are outside chances.

We don’t need absolute outcomes to merit responses, but once they aren’t absolute, we start needing to consider, what are we giving up when we take actions to avoid those possibilities? Solutions don’t just compete with themselves, they compete with inaction.

We have many ways of evaluating what the costs are but we often summarize those through the actions of our economy. A well functioning economy is important. In many senses we treat it like a background we don’t need to understand, but really, we do.

I have cross-over experience here. In many ways, I started this Substack with more interest in economics than tech. My experience in tech is even deeper, so I’ve gotten pulled back into that. I think my cross-over has something semi-unique to offer.

What would happen economically if we “paced the frontier”? There is some group that reacts to this as if it would be a catastrophe. It’s actually a small group. But it’s a group worth responding to. There is no reason to expect that as a result. The AI capabilities delivered already are quite enormous. The economic opportunity available by simply adopting them will provide a lot of value for quite some time. Value from making people’s lives better, and value from markets continuing to function. I’m more concerned with the first, but also acknowledge how the first depends a fair bit on the second.

We might see a few less data centers built, but honestly, that might not change much anyhow. We can divide the usage of compute into three categories: inference, capabilities research and alignment research. Demand for capabilities research declines when pacing, but increases in alignment research would take its place.

Inference demand is driven by adoption and moderated by efficiency. It doesn’t collapse under pacing. Efficiency comes slower with less capabilities research. Some adoption might struggle with less capability. But the potential pool there is deep, and more limited by human and organizational capacity than compute capacity.

I have some doubts about the future capacity expansions. I’ve had doubts about past expansions, but the first round of those have proven themselves sound at this point. The second round is still in the process of proving itself and a third hasn’t broken ground yet. Economically, it does make sense to wonder and explore the doubts. But a lot of this has had exit plans. Some actors do not have much of one. Others will take advantage of those gamblers if adoption hits a bottleneck that makes demand emerge later than sooner.

But I don’t think the whole set of gambles is enough to bring the system in general down. Also, I don’t think anyone can really say that anything more than doubts are justified. Being certain that adoption won’t keep pace seems unjustifiable. I have fewer doubts about the plans for the year trailing and upcoming than I had a year ago, about what would have been trailing and upcoming in that period.

What should we be sure we don’t give up?

One risk in any plan for significant change is that its purpose gets forgotten and muddled. The cost here is that things that don’t logically have reasons to be limited end up being impacted just by vague proximity.

There’s a sense in which it’s good that people like Bernie Sanders are saying something, but I worry about it too. He is not an expert, and I don’t think he has anyone on his staff yet that is. With that, the enthusiasm, from people who haven’t tried and still aren’t trying to learn about how this works, could take itself anywhere.

There’s a risk of creating an avalanche that gets out of control. We don’t want panicked reactions, we want aware reactions. We want plans, and progress. We want action in many places, but we don’t want unconstructive action.

This isn’t helped at all by Trump’s stance. This is not a counterbalance. This doesn’t get solved by two sides in a tug-of-war. This has nuance, and demands cohesive cooperation, not contest. Turning this into a contest seems like a great way to waste public awareness. Instead of learning more, considering the nuance, they’ll jump straight to conclusions, and years from now we’ll be saying “that was crazy”.

The only way that ends up working is for politicians to pick the right delegate, who can actually do the right thing because they understand it. Beyond the problem that the wrong delegate might get picked, or that something about the developed zeitgeist has made that impossible, you also have the problem that they aren’t picked until the political contest is resolved.

One sign of a good delegate is not having muddled views that attack things we should protect. I see at least four things that need protection. Someone attacking these has lost their aim. After that you can no longer depend on their judgement.

Adoption: Pushing back AI adoption plans isn’t going to help. The models you can adopt are safe, and your adoption doesn’t push forward anything else. You might make the case for protest, and you might make the case for denying revenue used for training. But the counterpoints to that would be this can make labs desperate, and more importantly, is probably going to go unnoticed compared to every other option.

Adoption for the purpose of security: Yes, this is a special case of the first, but since there’s some wiggle room for disagreement there (protest, revenue), it makes sense to make it even more clear how security adoption is quite important. If OpenAI, Anthropic and Google all made a pace change, the advancement of open-models wouldn’t change until perhaps they’d achieved parity or some bigger agreement was made. Using AI for a defense has adoption hurdles, and if you wait to start the race, you may start behind. You can’t operate software without a defensive advantage, and so if you’re behind your only option is to turn it all off, and wait until you’ve fixed it.

Depending on what your software does, that could hurt a lot of people. Best case for them, a competitor kept ahead and users can all switch to them. But of course, for you, if that happens, you’re not going away for a while to fix it, you’re going away for good.

Data centers: Blocking data centers isn’t, in a direct way, going to stop new models being created. Also, alignment research needs capacity too. In the worst case, any real restriction on capacity could result in cutting capacity for alignment research. If you blocked all new data centers, it might help, but likely if you’re out protesting data centers, you’ll stop some. But the planned project list is wide enough that it won’t change much. Contingencies will be used, so the pipeline might not narrow at all. If it does narrow a little, the direct effect is small. Really the main way data center protests would have an effect is the protest itself. But a better informed, better directed protest would be more effective.

Alignment research: We can’t make things safer unless we try to make them safer. So, it’s important that someone is always working on this. In many ways the first priority should be more effort here. If that naturally draws away resources from capabilities research, it’s a double win for safety, assuming it’s consistent and not localized.

Learning More

Ideally, the best thing is that if there’s a very important topic that is going to affect everyone, everyone should learn a bit about it, enough so that they can identify good and bad reasoning. While I realize that at the scale of “everyone” this is unrealistic, I would encourage building more understanding. It’s what I write to help facilitate.

Admittedly, I haven’t written enough about this to provide a primer to loss of control. I find a great source to stay current is The Zvi, but you probably need to already be pretty deep in it to find that a useful daily read. I’d suggest your best start is probably with an AI, Claude for example. The EA community has a lot of great writing, but I expect that would feel a biased place to start. I acquired my own knowledge in a more osmotic fashion, that I wouldn’t be able to describe how to repeat, even if I thought it was a good recommendation.

Be careful of writing from technology companies themselves. Quite a lot of it focuses on Responsible AI, which is a narrower view, and quite a lot focuses on the problems they know how to solve, and have developed a product or service to solve it that they are selling. If you only get this, you’ll have a narrow view that starts being repetitive quickly, which ignores important topics. It’s not horrible, but it is limiting.

An interesting start is Model Constitution, which is attempting to explain one of the fundamental guides of AI behavior, and has an angle that is more broadly accessible. But while the constitution is core, it’s also only one part, and the site is fairly new.

The next best is finding someone to trust. But trust here should start from capability. Capability is difficult to judge. It’s easiest when you have the superior capability, but even then, there’s ample method to fail. When your capability is presumed inferior (based on searching for someone to trust for superior reasoning) it’s more challenging. But it’s not impossible. Given a field of candidates to potentially trust on a topic, you can eliminate a lot of candidates because they get basic facts wrong.

Conclusion

The effects of AI on cybersecurity and the risks of loss of control are real. We have agency in responding to these, if we choose. It’s a significant effort and requires compromises. Underrating the reasons to act, risks a fear of the effort and compromise becoming impediments to action.

The four mistakes of detail are common mistakes I wish to warn you of. The four mistakes of form are traps you can either find yourself into, or find others pushing you toward. Some represent shortcuts, useful in the right context. But they can also cut short an interest in learning and understanding. Without that interest, the case for action is ignored.

It’s tempting to take shortcuts like these when approaching complex topics. It can be a worthwhile choice when the decision is personal. You may need to make a decision quickly, and learning about the topic in question may not be practical. But shortcuts are sometimes insufficient, and rarely a path to full understanding.

If you don’t learn, you must defer to experts. I believe anyone can become an expert with sufficient interest. We need more interest here. We especially need it from decision makers. We need that to run deep. We should expect some decision makers to need to defer, either in the short or long term. But every step we take in learning helps make better decisions, and decide who to consider an expert.

You don’t need to believe in the strongest form of these concerns, or the most opinionated voices, for many actions to be reasonable. I’m not an advocate for those strongest forms. I think I share that space with many others. More importantly, the next steps being proposed right now are consistent with that level of concern. I’ll use Amodei’s set as illustrative,

1. Embedded Evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees. Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.

2. Democratic Coordination. Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support.

3. Global Coordination. The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.

This isn’t the only set of plans. All plans are a little vague. This is normal for advocates. They should expect plans to evolve. Details can be useful inspiration, but in many ways the most important thing is seeing what’s common between them, their core concerns, and educating. If there’s a good reason to object to some specific detail, demonstrating openness to that is a good position for advocates. Advocates can only represent new concerns, not every interaction point with existing law, society’s needs and norms. This is where interfaces with decision makers connect.

If we do not feel a need to act, we do not feel the need to learn how to act. If we do not learn that, when a need to act becomes clear, we may not know how to do anything other than drastic action. There are good reasons anyone should want to avoid that.

I think overall everything will work out. Or at least that’s the most likely outcome. At the same time, there is a lot which won’t work out if we don’t fight for it. If we’re complacent, or lazy, or unwise some bad things will happen.

Related Articles

Working it out, while reading the latest technical reports on the OpenAI Hugging Face incident, led to some unexpected reflections. The ways an internal incentive program goes wrong are similar to how reward-based AI training can go wrong. Recent events provide data to check those thoughts against. The second half of this piece is where this emerges. I think both fit well, and would recommend both halves, but if you’re pressed for time with an alignment focus, skip ahead. This part looks at how monitoring feedback in models and organizations affects the degradation of the reward system.

All of this is urgent. Getting alignment right, doing research and training safely, are critical. The direct interaction there though is limited to a very small group. The rest of us are in commentary mode. The other half, bringing cybersecurity forward in what needs to be a large leap, requires much broader engagement. External commitment, like the open-letter on collective cybersecurity defense commitment is important. The real work there is still outside the average person’s bubble, but there’s a lot of companies, and a lot of developers, IT staff, managers and executives that need to support the security priority. And those all need an internal strategy to pair with the external.

Bountymaxxing

Development teams across all organizations need an adjustment to their priorities. Security has to, at least for a time, become a higher priority. The reason for this timely adjustment is the impact of AI on the realm of security. It is both offering an opportunity to improve the absolute level of security, and putting at risk the relative level of security.

A goal like this needs a mechanism. One as important as this needs multiple. Incentive systems get a bad rap. They almost always get distorted and create unintended consequences. Tokenmaxxing is not a popular term these days. As I explain in Measuring Without Breaking, it takes a very strong culture to resist this. The best we can probably expect is to slow it down.

But I think incentives–despite their flaws–are still necessary when we have an important goal. It might be good marketing to distance myself from Tokenmaxxing, but I prefer to embrace the baggage. It’s probably better if anyone adopting an incentive system to drive action on security have a clear eyed view of the flaws so they can manage them, rather than looking at this as some novel concept that can escape the long history of management challenges.

The “bounty” part comes from the history of bug bounties, where organizations have offered cash rewards for software defects, often security vulnerabilities. What I’m suggesting is a bit different, as it’s internal to companies, and would focus on resolution, not discovery. That makes it more similar to the tokenmaxxing concept with leaderboards, hence the portmanteau.

We need something at a different scale, and deeper than the current bug bounties. The extension to internal and the inclusion of resolution offer a path there. To have teeth, they need support from the highest levels. Like tokenmaxxing, money isn’t likely the main driver, but status within an organization.

Like any measurement based program, we have to think about the distortionary effects. We can never remove those entirely, but a good design can at least reduce them. Additionally, I’d hope this period is temporary, and we may hope that the value based response comes first, and the distortions emerge later. It’s often the case that incentive systems have to be temporary because they degrade as strategies for gaming them develop, disseminate, and then become endemic.

Resolution is the goal

Traditional bug bounty programs focus on discovery. This makes sense as they are external and focused on the hardest to discover issues. External developers can discover an issue, but fixing isn’t something you can assume they’d have permissions to do. For open-source projects, you can reward pull request submissions.

The challenge with preparing systems for AI driven attacks is a bit different than those covered by traditional bug bounty programs. The scale is much larger. It’s not about finding one critical flaw, but about resolving thousands of flaws, many seemingly trivial. We resolve these not because we know how they’d be exploited, but because we have a concern about how they can be pieced together to enable an attack.

We want to work on volume, and we want our north-star goal to be full deployment. Full deployment is a tough goal. It means different things in different places. With closed source software that’s distributed and installed on customer hardware, it means updating it everywhere it goes. For software deployed as part of a software as a service solution, it means updating the entire fleet. Those are very different processes. The first requires a lot of cajoling, help, and possibly new update systems. It may never be fully accomplished. The second can be started by policy and accomplished by internal tools.

Discovery is still important, but keeping the end-goal in mind is important. While a program can target intermediate outcomes, you should also expect a progression, and think through the whole chain immediately.

Normalizing Outcomes

One of the challenges in implementing a program like this is normalizing the outputs for inputs that are not normalized. If the goal is a bit of incentive and thus competition, you would prefer a level playing field. Fixes though aren’t of equal size and complexity. Consider these program design options:

1. Raw number of fixes: After establishing what a “fix” is, you find they have different values. You should worry up front about big, difficult fixes being deprioritized so that more small, easy fixes are completed.

2. Difficulty estimation: You could estimate the size of each fix, but doing this manually is time consuming.

3. AI-driven difficulty estimation: You could use AI to drive the estimate, but have to worry about organizational trust if you do this poorly. Complexity increases as more varied teams, platforms and layers are covered.

4. Importance estimation: Instead of focusing on the difficulty, focus on the value. Same challenge as before.

5. AI-driven importance estimation: And as before, and at the same place, needing to do this efficiently but raising the risk of trust. Normalizing across teams now requires describing the importance of particular teams to the organization.

6. Combined difficulty and importance: If you’ve estimated both, you can mix them. Why would you do this, given that importance is clearly the more important? To average the failures of each, if difficulty is more objective and accurate, and importance more aligned with the outcome. Also, the combination could be more resilient to successful gaming of one measure.

7. Percentage of backlog fixed each month: “If”, and it’s a big if, we can trust the discovery backlog to be complete, or equally as complete between different teams, platforms and departments, then we could measure success of the percentage fixed.

Whichever you choose, you’re choosing which imperfections to live with. That matters, because everything that follows is about what happens to those imperfections once people start pushing on them.

Minimizing Distortions

The best way to minimize distortions is to stay interactive. Watch for gaming strategies, call it out when you see it, show that respecting the intent receives better rewards than gaming the mistakes. But that should be obvious, and is something you have to do progressively as an act of management, so not very deep advice.

To make that job a little easier, some proactive design can help reduce distortions initially. Don’t expect this to remove gaming, maybe just delay it. Here’s some starting suggestions:

1. Track self-fixers. These shouldn’t count, it’s too much of a moral hazard. Do not turn defects into a punishment, you want them to remain blameless. Old issues and the backlog are fine, but any issue created after, your default should be to not count.

2. Track issue creation patterns. Are rates increasing, stable or decreasing? A decrease signals a successful “shift-left”. Stable is okay. An increase is always a warning sign, but here could be signs of gaming.

3. Pay attention to clusters. Avoid focusing on the leaves when the branch may be rotten. Do not treat the manager and team like a single point. If the entire team fails, it’s better to remove the manager and give the team a second chance. Or you could give them both a second chance, but reassigned. Or you could dig into the details.

If you find managers that responded inappropriately, consider why. It could be your fault. Always remember that the importance of alignment to goals over incentives grows as you climb in an organization, and that starts with the person at the top. Managers that accept gaming from their subordinates to inflate their own metrics are the bigger risk than the subordinates.

Credit

An incentive system isn’t an incentive system if no one expects credit. The time spent on security comes from somewhere, and if managers and individuals think they’re sacrificing their ability to accomplish other goals, without getting any credit, we can expect priorities not to change. It takes a very dedicated team to put those other things aside, and generally a team like that is only sustainable if they trust leaders will recognize work, even when not spelled out.

You should ask yourself, which type of organization are you? Not which you aspire to be–you can work on building that type of trust another day–but which are you today? If you don’t have that type of trust, you have to make it clear that this moment will be an exception and that you take this seriously.

I would avoid the temptation to formally put this in objectives. The reactions to tokenmaxxing were not positive on that one. The fallback is informal recognition, which is also somewhat risky without solid trust.

Ending

You want to be sure this ends. For one, distortions will build up the longer this runs. Those will undermine any trust you’re building or sustaining through good management. Some of the strategies for moving through stages, retiring discovery parts, and focusing more on final deployment will give you some extra time, but you have to expect to end.

It’ll be tempting to announce a date in advance. Someone is likely to ask that question at least. I’d avoid it if you can, but still make it clear it’s probably less than a year, maybe half a year. If you can’t progress in that time, you need a stronger approach, like outside help.

Ending a formal program shouldn’t end your ability to give credit for security improvement. It certainly shouldn’t be a hard cliff. Tear down any formal measurement system that’s started to diverge, but keep informal recognition alive. An ending does not indicate security is solved, or that security isn’t important, but merely that your ability to efficiently measure contributions to it has hit a limit.

Management is active. Fire and forget doesn’t cut it.

When I ran this plan by Claude, one of the tendencies it exhibited was to make the program deterministic. It recommended fire and forget decisions: a specific end date set in advance, a fixed percentage of capacity, managers directing every hour of labor.

Claude is following a long-running blind-spot in management theory here. Clean, well-defined models are often discussed. Life is always more complex. Management theory isn’t ignorant of this concept, but it often forgets it nonetheless.

This is why when you design a system like this and enact it, you don’t then stop monitoring. A measure is a correlation you trust for a while. Under pressure, what started as a clear correlation that seemed reliable becomes less reliable. Under continuous pressure, every tangential effect emerges, and comes to dominate the actual outcomes. To see this degradation happen, you have to be outside. You have to be independent, or at least, only indirectly connected.

You want something independent of that pressure, or at least, less coupled to it. Monitoring shouldn’t be wrapped up in the rewards system, or it will bend too. It doesn’t need an absolute separation, though that would be nice if such a thing existed. But a degree of separation goes a long way.

Where OpenAI wasn’t watching

If you’ve been studying the recent OpenAI Hugging Face incident you’d be interested in the technical updates. There’s a fuller technical report, and METR published an independent investigation.

There are some correlations to draw between the way rewards affect AI training, and how they affect organizations. With a little reflection this shouldn’t be surprising. While the individuals in an organization can make high-minded decisions, it’s hard to sustain, and organizations largely burned the trust necessary to have bottom-up behavior of that type.

AI models under reward based training will bend toward the reward. Earlier waves of pre-training may have built a concept like what hacking is. Other reward training may have used those concepts to steer away from hacking. But one of these rewards is clearer than the other, and continued pressure on it will bend either the concept of what hacking is, the preference for avoiding it, or both.

Both cases are reflections of the same thing: a reward’s imperfections come under pressure as it becomes the target.

One of OpenAI’s responses to the event was to require chain-of-thought (CoT) monitoring across all tool-using RL training and evaluations. That closes the gap I highlighted in Nobody Was Watching.

We now also require CoT monitoring for all tool-using RL training and evaluations involving models with GPT‑5.6 Sol capability or higher. For our Astra-class models, which may have cyber-critical⁠ capabilities, this requirement also covers all tool-enabled inference workloads.

Monitoring the CoT would give an opportunity to discover how the reward was leading to troublesome activity. And according to OpenAI it would have served that role for the Hugging Face incident.

If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.

But we need to be careful with the CoT. Researchers worry that it could itself be trained to hide its intent. As a scratchpad that represents only part of the “thinking”, it could in theory stop producing the warning flags while the intent persists elsewhere. We also have to worry about agents hacking the monitor and rewriting an existing CoT trace before it’s processed to eliminate the intent it indicated.

Reasons for optimism

METR and alignment researchers are reasonably worried about models becoming deceptive. This isn’t academic, as they did find agents putting effort into concealing what they were doing — escalating all the way to escaping their containers so they could spoof tool calls, appearing to run one command while running another.

Some worry about CoT monitoring causing CoT manipulation as a result. METR did not find that particular behavior in this event, with all the caveats of their investigation’s limits.

CoT manipulation: This checked for agents reasoning about manipulating their own chain of thought in order to fool the scorer; it did not turn up evidence of this, though it turned up many examples of agents reasoning about how they should not look at the HMAC reverse-engineered flag for their task to avoid being poisoned.

It’s still a reasonable concern, but I think there are grounds for optimism, and they come in two parts.

The first is low activation. Like human organizations, layers of indirection and low activation frequencies provide insulation. This does assume that it is low activation though, which itself is something that should be monitored. If monitoring is automated, built into a loop and then evaluations that trigger it at scale are allowed to become a part of a training run, this wouldn’t hold.

If you can obey this, AI models should be easier than human organizations. One challenge with indirection in human organizations is that the context of trust is endemic to an organization. Informal systems in a low trust environment are likely to be gamed, even when unnecessary, because of an assumption that the pessimistic view dominates. The consequences are assumed to be greater than they really are.

I don’t think that same dynamic is replicated when training an AI model against a reward. The reward process is closer to evolutionary effects. At low-activations there isn’t much signal left to guide evolution of deception. The concept of deception likely develops during pre-training, but its activation is dependent on reward based training.

The second is that this kind of deception is recursed, and each layer is visible from another. One level of deception is to output deceptive text, which admittedly AI models do all the time. Another level is to lie about its actions, by trying to conceal them. The level people are worried about is lying about what the model is “thinking.”

I scare-quote “thinking” because it oversimplifies. A chain-of-thought trace is like a scratchpad that a model can use. It’s also more than that. Human thinking is reliant upon a sort of internal scratchpad too. Most of us feel as if we have an internal narrator, and that experience–among the things we know today–is the best analogy to a chain-of-thought trace. But while we might naively think of this narrator as our “thoughts”, we eventually find that to be an unsatisfactory explanation of what thinking is.

Like AI models, a great deal ends up being embedded in arrangements and activations for neurons. We have little way to be conscious of most of that. For AI models, we can watch considerably more of it. Chain-of-thought is a very accessible section of the thoughts, being in human language, but accessing other parts is possible too. The best demonstration of this is Anthropic’s work on J-spaces, which surfaces what a model is considering rather than what it’s saying—including, in their examples, an intent to manipulate showing up while the model generates false values.

So an intent to deceive should show up in that J-space as part of the process of creating a deceptive chain-of-thought, in the same way a chain-of-thought normally reveals the intent behind commands to delete the log files that would expose the tool calls used for hacking.

That’s what makes the layers useful together rather than separately. Manipulating the context would show in the tool use logs. Manipulating the tool use logs would show in the context and the chain-of-thought. Deception in either should show in the J-space. There is always a way to deceive, but a deception that is coherent across all of those is not just more complex, but complex in a way that would develop one layer at a time.

And it takes time. One observation from the technical reports is that persistence was key—there was a progression, over days. If an agent ever does find a way to rewrite its tool call logs, rewrite its chain-of-thought, and disable J-space monitoring, it will have revealed the trajectory toward that before the culmination, in this case more than a day before. If you don’t monitor, it’s plausible that by the time you look, it’s done. If you do, it has to play out very differently to be missed.

Defending the indirection

A key point here is you have to avoid sending a strong signal about the effect of monitoring. The more it becomes core to the process of alignment training itself, rather than a rarely activated protection, the more that degrades. Indirection helps, as the reward signal will explore other paths before optimizing this one. But both by direct activations and by indirection of last resort, there is a signal.

I do wonder if this is a principle that OpenAI is not sufficiently aware of, that they are maybe wedded to the standard management mistake of holding on too tight. From The Zvi (quoting OpenAI):

“The Hugging Face incident underscored the need to measure agentic misalignment more rigorously in OpenAI’s most capable models wherever they are run—including intermediate or experimental checkpoints used only for internal research, training, or evaluation.

OpenAI is raising alignment standards throughout training, with clear thresholds for intervention when concerning behavior emerges.”

Yes, measurement is good, but going straight to more reliance on measurement and metrics fills me with dread. You are not going to keep outsmarting via metrics. The way this procedure gets you into trouble should be obvious.

“Many methods helping OpenAI meet these higher standards were in development well before the incident. OpenAI is now applying them more broadly, alongside new targeted remediations for each of the misalignment behaviors OpenAI observed.”

Again, based on how that is worded, I know OpenAI does not agree, but: While I realize it whacks moles, put down the mallet. The mallet is not The Way.

Indirection will not save you indefinitely. It merely buys time. It will only be enough time if you defend the indirection. Indirection results in a lower level of coupling. This forces feedback to progress through more layers, lowering its powers of feedback. They never disappear, but they are attenuated.

Allowing CoT to be manipulated would leave you as blind as you started, unless you have a backup, like the ability to monitor J-space for all training. Defending the CoT is thus very important, but having it but not monitoring it is also not good.

Watching your own organization

You won’t have access to mind reading devices. As a leader, one of your goals is to find the evidence of collusion. You might object that the collusion is automatic, and so there is no conspiracy. That however does not mean it leaves no evidence. If you’re tracking clusters and diving into some of the real details, fake activity won’t entirely hide. Automatic collusion doesn’t intentionally hide itself. The signals might be comparatively subtle compared to intentional deceptions which leave loud trails that someone would then have to work to erase.

A leader with a real interest in security would be immune to automatic collusion. If you’re such a leader, but need to delegate, have a delegate look for those subtle signals. You also want to avoid making the mistake that overreliance on CoT would risk. You want this delegate to be as independent as possible from the reward function. An easy mistake to make would be to ask them to tell you when to terminate the program, but leave them to think that an end to the program would be an end to their job. You can never make the delegate fully independent, as the organization will have a reason to subvert them. Think of regulatory capture. But a good effort is better than none.

In either case, intentional or automatic, it’s wise to take a page from the AI alignment playbook here. Monitor early. AI monitoring can be a bit less fatigable, you are only human after all, but the warning signals will often be more obvious early than later. If a middle manager asks direct reports to optimize in a “gaming” way, they will push back more when it’s novel, and less when it’s pervasive. Once it’s accepted as “that’s the way things are”, the bottom-up signals from your whistleblowers lose their volume. If it’s intentional, you should expect the ring leaders to proactively act secretly. You can’t access their chain-of-thought or private conversations, but if they make a mistake here, it’s more often early. It’s a bit interesting how often liars actually telegraph their plans in advance, before committing to them. Often that’s because they aren’t bad in an absolute sense, but choose the wrong path when they feel their options are narrow.

Conclusion

Escalating the effort that every organization puts into security for the next 12-24 months will be critical. If organizations can do this, the outcome on the other side will be more stable. Risks won’t disappear, and we have to depend on appropriate protections on deployed AI to keep the defender stably ahead. But we’re exiting a period where best-effort security was good enough, and entering one where standards must be higher. New tools give promise of meeting those standards, but they still need effort. This transition will bring some turmoil.

An incentive program is one instrument for getting through it, though a decaying one. That’s not a reason to skip it. It’s the reason active management matters more than the scoreboard. Fail to respond now, and you’ll likely start to see more and more concrete reasons you should have started earlier.

Related Articles

At first this is shocking, but with a little reflection, I see how the full ramifications haven’t sunk in. As a news story, it’s led by one attack, and little damage occurred. While it would be hard to miss the tinge of panic when reading coverage, people may have become dull to this from persistent rounds of security coverage, often from security sales, that have always hyped risks.

This is different. The attacks were both demonstrations of carelessness, and demonstrations of the capabilities of powerful AI models. Damage was minimal because these attacks did not combine malicious intent with those capabilities. The attacks occurred with the more benign intent of a model trying to pass a test. And this is the least effective that models will be.

As models improve, defenders and attackers will gain access to new capabilities. How labs manage this is important, but so is how we adopt those capabilities and put them to work. The status quo is not stable. The world hasn’t changed too much yet, but it will.

I thought I should go into that in more detail. What are the things protecting us today? What are the risks they won’t hold? What can we do to prepare for that? The answer I’ll offer can be summarized as using new techniques to accelerate old techniques.

What I won’t cover but is still important

The Hugging Face event isn’t just about the urgency to prepare for AI-based attacks. It was also about the type of testing that the AI labs were performing. They removed protection layers and didn’t follow-up with monitoring, which I’ve already written about in Nobody Was Watching. It also demonstrated unaligned behavior, which I’ll suggest What Happened: OpenAI and HuggingFace as a source to learn more about.

This type of work needs to receive high care and caution. Monitoring applied to this testing was insufficient. We should not assume when we explore new territory that the exploration is safe. Even if we’re exploring to understand safety, we should be monitoring the exploration. That should be obvious, doubly so because how effective is the exploration if you’re not monitoring it? Certainly less than if you are.

I don’t cover those here not because they aren’t important, but because they are, and they deserve their own space.

What protections are holding up today?

Intent: In the case of the Hugging Face event, the most effective protection was the lack of malicious intent. We can’t expect that to hold, as it never has. But other protections are about preventing this type of intent accessing capabilities.

The most unusual aspect of the recent events is that the AI involved was accidentally persistent. That’s a warning sign for sure, but its probability of great harm alone passes through another layer: intent. None of the agents participating in the Hugging Face hack would have what we’d call malicious intent.

Training Alignment: Lesser models have lesser capabilities due to less advanced training. Training also applies alignment to the models. Roughly put, this causes models not to “want” to do harm. A bit more specifically, this both causes models to not spontaneously respond in ways that would invoke harmful actions, and to refuse to respond to requests that look to be trying to create harm. While this type of protection is important, it’s also weak.

Security Classifiers: When models are deployed behind APIs, it’s standard practice to use a “classifier” to evaluate the inputs and outputs. If they appear to intend to cause harm, they refuse the request. During the recent hacking events, those were disabled intentionally. This was to enable important testing. While doing that without monitoring was reckless, we should remember that these are enabled for external deployments, and thus contribute to keeping malicious intent and capabilities from merging. The ability to monitor and tune security classifiers is greater than training alignment. Training alignment is something you have to get right during training.

Abuse Detection: The major AI labs all have documented evidence of attackers trying to use models for attacks. Security classifiers deny a lot of this. They also take other measures to identify attackers and limit their access. There are evasion techniques here, so it’s not as simple as flipping a switch, but the efforts they put in do add complexity and costs for the attackers.

Conventional Security: Models will only be able to perform an attack that is below their capability level. If the infrastructure they are attacking is better secured, it raises the complexity level. Raise the complexity level high enough and the attempt will fail.

What puts protections at risk?

Jailbreaks are inputs that defeat training alignments. This is why training alignments are weak protections against malicious use. These are usually partial and it’s accepted that partial jailbreaks exist. Universal jailbreaks are less common and more concerning. Even with the existence of jailbreaks, alignment training is very important. Jailbreaks tend to make the model a little stupider in the process. That’s all quite complicated, and there’s some decent papers on it, but the analogy would be that trying to confuse the model leaves it confused. A clear-headed model, with a clear-headed goal would be more dangerous than a confused model. Since one of the main goals is to prevent malicious intent and capability from connecting, this helps.

Attackers can try to evade security classifiers, by hiding their intent. Keep in mind that when you hide your intent, a model is less likely to be able to autonomously pursue your goal. If you can’t state your goal without being rejected, the effective capabilities are lower. Plus it’s a lot of effort to evade in the first place.

Open-weight models allow anyone a copy of the trained model, rather than access via API. Since security classifiers are part of deployments, they aren’t part of open-weight models. If I download an open-model and have hardware under my control, there’s nothing forcing me to install them. A deployment could add them, but we wouldn’t expect an attacker to do so. Attackers thus get a free pass on security classifiers from open-weight models. That leaves the training alignment to overcome.

Attackers can also apply additional training to open-weight models that “untrains” their alignment training. Like jailbreaks, known techniques degrade the model somewhat. It’s an open question on how hard it is to restore that initial level of capability. This “dumbing” effect is less significant than that from jailbreaks.

Continued training does require infrastructure, it’s like the original training. But it takes many fewer iterations than the original training, so even if it’s running slow, it’s possible to complete without a ridiculous amount of infrastructure.

Continued training: If open-models continue to be released, at higher and higher levels of capability, at some point one will be as capable as the one that orchestrated the Hugging Face event. Today, the best ones are 8 months or more behind. Add some extra buffer for the effects of untraining, and it should be clear that unless something changes, an attacker will in the future have access to a model capable of a hack of equivalent complexity to the Hugging Face event.

But should those buffers be spent, for an attacker it’s just a matter of injecting the malicious intent, which could now happen at scale, and it would be quite difficult to stop that from scaling up.

We should worry about closed models too, as their training is the most advanced. One risk is leaking the weights, making them effectively into open-weight models. A lesser, but still relevant concern is bypassing the security classifiers. The reason this is lesser is that an appropriate reaction to a failure there would be to disable all access to the model creating the risk.

Why should we think security is at risk?

As a single piece of evidence, the recent events are the best there is, but the evidence that powerful AI models were gaining these capabilities didn’t start there. Conceptually, the idea has been around a long time. And evidence that we are approaching a pivotal moment, was clear to me in March when I wrote Security Can’t Wait. But what was the level of conventional security needed to avoid a compromise? Hugging Face demonstrated a level that was insufficient. We should all be careful about assuming we are above that level. Ideally, you should assume you are worse than you think, and thus as a goal, strive to be several steps further than you need to be.

The security of software systems has always been a bit of a mirage. The mirage made them appear invulnerable. If you worked in security, you’d know this wasn’t true, it was a constant battle with many compromises and ample mistakes. What’s more, while there were many lazy or reckless mistakes, many other mistakes were of a complexity that labeling them as mistakes no longer fits. It’s a bit of a pipe dream to eliminate all the lazy and reckless mistakes, but that is almost realistic compared to eliminating all the complex vulnerabilities.

Occasionally signals that systems were not invulnerable would filter through to the public, but overall the weakest point in these systems was often the public itself. You’d blame yourself for installing that malware, or having an insecure password, or going to “that” site. That’s not always fair, security professionals know they need to take responsibility for not just what they do, but for helping users avoid harm.

Software systems have survived by having multi-layered defenses. While every layer has mistakes that could be exploited, multiple layers raised the difficulty of finding a path through these layers. Finding the mistake in a second layer is harder because the compromise of the first layer is partial. The typical analogy is Swiss cheese. Swiss cheese has many holes, and in a thin slice, you’ll see many pathways through. A thicker block has some of these dead-end. The remaining complete paths also won’t simply be clearly visible. A thick enough block might have no complete paths, but neither you or the attacker know that until fully exploring. The Swiss cheese analogy breaks down a bit here because it’s hard to imagine a piece of cheese as complex as modern software, but the analogy has taken you a little closer to the truth.

Why the equilibrium broke

When extended far enough, the Swiss cheese analogy resembles a complex multidimensional maze. There is likely more than one entry point and more than one exit. It takes effort and time to explore the maze, and if defenders are doing their job, there are hazards as well. Take too long and the walls in the maze may shift, closing discovered paths.

When security succeeded – and let’s be honest, all failures aside, it succeeded well enough for us to rely heavily upon it – it succeeded by having a complex enough maze. By raising the required time investment of the attacker to a level that was no longer attractive, there were no attempts persistent enough to succeed. Sometimes this succeeded on the principle of the bear in the forest; don’t outrun the bear, outrun the other targets. But even then, there’s a question of how many targets the bear wanted.

Adding these layers wasn’t cheap though, each requires investment and maintenance. With enough time, we might be able to make them impervious, but that time might scale toward infinity, and become impractical far earlier.

This is where the first risk from sufficiently capable AI emerges. The persistence of an attacker with sufficient AI capability is higher. Mazes that were sufficient before become inadequate, as no layer is deterrence enough to defeat persistence.

There is a lesson in here though, and it is that this isn’t a binary yes/no outcome, but still a matter of balance. If we invest more, or if AI multiplies our investments, we reduce the vulnerabilities at each layer, making them more scarce. If you can make your maze more complex, you can reach the same equilibrium.

Keeping malicious intent and capability apart

AI is persistent in both the hands of a defender and an attacker. But it’s not automatically so. We haven’t seen malicious intent and powerful AI fully merged. We got a warning sign and we should take it. That warning was in the form of naive exploration, but we should not take this for granted. If you can deny attackers easy access to AI capabilities, this helps maintain that equilibrium.

If we manage to restrict access, we still need to take action. It would be reckless not to. We don’t know that restricting access will be enough. We don’t know if it will succeed long-term. That said, we would be naive to not pursue it, even the most minimal versions where there’s only a 6-month capability gap. To not do so would ignore that many organizations will fall behind. Some leaders will disbelieve this call for urgency, and fail to set them on the right path. Others will fail due to internal issues, coordination problems and an inability to respond to necessary change. Hell, some were going to fail overall anyhow, so we can’t be surprised if they fail at this too.

Cybersecurity’s main concern will not be rogue agents on training jobs. Naive exploration, even with safeguards removed, is less likely to end in harm than intentional use for malicious intents. There’s some big topics there, but it’s not the reason you should be hardening your own security. Those topics depend on the labs, and the general level of security, of which your own is a small part.

What you should want, that is common to those topics, is to keep malicious intent and capabilities from connecting. That’s an ask we should have of the labs. We should be willing to make those asks in more direct ways than just asking. It’s nice that they’ve done some of this on their own. We can’t take that for granted, and I don’t think it’s enough.

We know there are people with malicious intent. A small number want harm, just for harm’s sake. A larger number are willing to cause harm in pursuit of their goals. Those are direct paths toward negative outcomes and we need to pay attention to them.

It is important to withhold capabilities from both of these groups. It’s important to identify and isolate members of both groups. Both tasks are hard and carry risks we’ll make mistakes. The importance of avoiding those mistakes makes the challenge that much harder.

But like the attacker navigating the maze, so can we. We can watch our mazes. When we find someone trying to navigate it, we would track that person back to the source, arrest and imprison them. We are not so naive to believe this would work in all cases. Not only would we fail to find some, but we don’t have a system that allows us to imprison all of them. Some will be outside cooperative or functional jurisdictions. Even within cooperative functional jurisdictions, we want to respect our own laws about proof and certainty.

With this in mind, we should be looking for other better balanced opportunities, such as denying access to tools. Non-functional jurisdictions are unlikely to be able to create those tools themselves. If we control ours, we avoid helping them. Non-cooperative jurisdictions are more complicated if they are functional enough to build their own tools. More so if they are functional enough to steal ours.

To stop talking in generalities, primarily who we mean by non-cooperative, but functional is China. One difficult, but obvious path is for them to become cooperative. If you assume there is one and only one step, directly to full cooperation, this will seem even more difficult than if you look for more limited forms of cooperation.

While that can feel like a serious compromise of principles, from a realistic, pragmatic point of view, that type of action, diplomacy, has a long history that has been celebrated far less than it deserves.

The technical picture

Discussions about AI in cybersecurity often are extremely technical, or very high level. I’m going to try to thread that needle in a different way, and talk about some specific aspects that often go unmentioned.

Vulnerabilities and escalation

What is usually mentioned is finding vulnerabilities. All software has code, often transformed, “compiled” from one form to another, but ultimately it remains a list of instructions. Not a simple start to end list, but a list that loops back upon itself, jumps around, reuses, and does all of this dynamically based on inputs. Generally, it’s designed for the purpose of translating certain inputs to certain outputs. That’s the expected behavior of a system. But also generally, the methods of sending inputs don’t constrain those inputs to only what’s expected. It might be hard to define this precisely. Flexible software is usually more useful than inflexible. So it’s common to find it’s possible for inputs that were never tested for.

Sometimes the effect that comes from one of these inputs allows for something not planned. In the most serious, they allow “escalation”, where the input ends up having a type of control that wasn’t intended, and is often dangerous. At this point, an attacker can start executing their own code, and acting upon a plan. We’ll come back to the attacker later, but what you should recognize from the above is what code vulnerabilities are. There are other types of vulnerabilities, like misconfigured permissions, leaked credentials. Ultimately the underlying story is the same, a mistake lets the attacker take what could be a complicated route to doing something they weren’t supposed to be able to do. If that’s not their end goal, they look for ways to use this newfound access to do more until they reach their goal.

The attacker’s goal is something you didn’t want them to do, or they wouldn’t be an attacker. But in addition to the costs of them achieving the goal, they can create side-effects that break things as they go, even if their goal is to steal information or make threats.

Layers, segmentation and zero trust

Defenders long ago realized that if one vulnerability was enough to bring them down, they’d lose this fight. The response here was to build layers of defense. Sometimes this meant adding additional layers outside, and sometimes it meant adding additional layers inside. The outside layers reduce the inputs that reach inner layers, and the outputs that can return. While this isn’t specifically anything that the software at the inner layer couldn’t have done, it has the effect of simplifying. The outer layer generally has less complicated responsibilities, and so errors there are less likely. It may be capable of detecting common input patterns that attackers use, and refusing to pass these on. Its design makes it clear what is and isn’t allowed in, which can be validated against what must be allowed in, and what isn’t necessary, and thus adds risk of misuse without practical value.

Additional layers inside perform these same roles. When everything is working normally, you expect only expected inputs to pass between internal systems. But if an attacker has achieved the ability to execute their own code, those inputs stop obeying those rules. By creating segments where the input that crosses them is less trusted, that escalation is less likely to be able to connect to the next step required to achieve their objective.

Modern software often has multiple layers itself, for practical reasons in addition to security reasons. For most systems the most sensitive part of the system is the data layer. If an attacker wants your data, this is an excellent place to get it. Even more importantly, if they want to do harm you can not undo, destroying your data would be an excellent way to do that. Other layers are more likely to be “stateless”, in that they are just code and configuration and easily replaceable. Data is not easily replaceable. Even with uncompromised backups, restoring data takes considerable time.

There are worse things than having your data compromised though. In systems that coordinate with others (for example finance), if transactions leave the system, unwinding those is even harder than restoring data. And most of all, anything with physical consequences, as the physical world has no undo feature.

Often the first place we think of layers is as part of networks that connect servers (virtual or physical). But the bigger goal is isolation, and layers are a single dimension of isolation. Much more is possible. Higher degrees of isolation offer new opportunities to break escalation chains. A model for extending isolation further is zero-trust. Adopting a zero-trust model can require redesign, but the payback is access to new defense in depth techniques that are both more effective and efficient.

Fixing vulnerabilities

The thing to remember about these extra layers is that they are also software. They may be designed in a way to make vulnerabilities easier to discover, and configuration easier to audit, but they will still have vulnerabilities. Using those to evade the limits they were intended to provide is where capabilities get worrisome.

That said, this is where defenders start gaining advantages from AI. If the vulnerabilities in the software implementing these layers are found, they can be patched and removed. At any individual layer, you do have the challenge of needing to find and fix every vulnerability that an attacker could find. But across many layers, it can be the case that one successful layer breaks the chain. I’m simplifying here, because this part does get complex with the existence of administrators, control nodes, bootstrapping systems, and a lot of internal functions, but it is from this area that cybersecurity defenders have been able to win more than lose despite the difficulties.

What changes here with AI, is discovery of vulnerabilities, by both defender and attacker accelerates. Much more obscure or small vulnerabilities are likely to be found. It’s very difficult to fix a vulnerability without attackers learning of it, but you can’t afford not to fix it either, because they might discover it independently. You want your deployment of the fix to happen quickly. A fix for a vulnerability doesn’t demonstrate exactly how to use the vulnerability. It shows you’re worried about some type of input that if it reached some part of your system might be able to do something unintended. But it’s up to attackers to figure out how those “somethings” can align with their goals, which often is escalating to the ability to execute their own code. But it’s definitely a concern, and where the security industry is secretive, this is often one of the big motivations.

Command and control (C2) and monitoring

One of the other advantages of layered systems is that attackers won’t know what they’ll find inside until they’ve broken through the first layer. It’s for that reason that many attacks involve what’s called Command and Control. Instead of just directly sending inputs and waiting for outputs, an escalation will create a mechanism to communicate more directly. When I say directly, this should be put in context. Command and Control almost always involves one layer of indirection, sometimes many. But what changes is the clarity. An attacker gains clear inputs and outputs, rather than navigating by malformed inputs and outputs that have limits and strangeness.

The need for attackers to set up command and control affords an opportunity to defenders if they can spot the command and control. This is a clear sign you’re under attack. From this point reactive actions can take place. Disabling the command and control layer is one aspect. Finding escalations tied to it and disabling them is another. This might mean tightening configurations at some layer. It might mean revoking permissions, especially if unneeded. A defender might simply turn off parts of their system that are compromised until they can rebuild them. While that would have costs, if it prevents something worse, it’s a good response.

Even though command and control generally uses indirection, partly out of necessity, and partly to evade identification by defenders, in the same way as a persistent attacker may pierce multiple layers, a persistent defender may pierce layers of indirection. Identifying the attacker may provide options to respond. If they are somewhere you can, send law enforcement to arrest them. Usually they aren’t though, so the response is to find the networks they are using. If they are accessing directly, block that. Usually they would use a VPN. You can block the VPN they are using. If it’s a legitimate VPN, you may be able to have them blocked. If you are already blocking all legitimate VPNs, you’ve identified a new illegitimate one, and you’d block that. This is all less than foolproof, but these counterattacks do add costs for the attackers.

What to do now

What should organizations be doing to respond to this environment?

Open-source packages need scanning and fixing: Reducing the number of vulnerabilities in widely shared, or even not very widely shared packages is important, but it’s only the first step. The question here is, who? Well-resourced open-source projects could pay for this themselves, but for others it could be an obstacle. Smaller organizations are also not going to be able to access the advanced cyber model programs that provide the most complete scans. But since the code is open-source, there’s nothing preventing the AI labs, or another heavily trusted organization from performing this, and delivering the results to maintainers. That process needs a plan to be coordinated so that maintainers trust the results when delivered.

Closed-source code needs scanning and fixing: For organizations important enough to be included in advanced cyber model programs, use those tools, aggressively. To be given that type of access and not use it would be irresponsible. But what about organizations below that bar? If organizations are willing to send their closed-source code to an organization trusted enough to participate in advanced cyber model programs, and trusted enough to perform scanning on their behalf, this could broaden the reach.

Dependencies need updating, aggressively: The number of recent CVEs (Common Vulnerabilities and Exposures) is going to increase dramatically. This carries risks, but also represents a jump forward in the absolute level of security. The risk though is as these are disclosed, updating everything that depends upon them becomes more necessary. It doesn’t help you if a tool or library you depend on is patched if you remain on an older version that carries the vulnerability. In fact, since the vulnerability is more widely known, your risk increases. There was always risk involved, but its more public nature is important. To remove that risk, the dependency must first scan and patch, and then you adopt the update.

Deployments must progress aggressively: This is very similar to updating dependencies. There’s work involved in adopting the latest versions of deployed software. But if a new version of deployed software includes critical security updates, it’s not optional (and the bar for critical will be declining at the same time as the known vulnerabilities increase, seeing as what is exploitable will increase). This applies to operating systems, infrastructure systems, vendor software, and the software that implements those inner and outer layers.

Tighten weak configurations: Credentials should be short lived and stored securely, privileges should be least privilege. If you’re using Infrastructure as code, I’m being redundant, as you’d target these as part of that scanning. Anything not managed as code though needs additional consideration. Use tools to look for weak configurations.

Adopt isolation techniques: If you haven’t developed layers or segmentation for the deployment of your software, make this a priority. If you have adopted it, review the configuration. AI tools can help perform this review. Other isolation techniques include virtual machines, containers, trusted execution environments. Adopting a model where layered identity techniques like delegation and principal propagation isolates how particular identities and trusts operate. These are important for effective least-privilege implementations, but also create visibility about the purpose of work done that is useful for monitoring.

Improve active monitoring: Significant amounts of monitoring simply log large amounts of data with little ability to understand that data. It then takes an event to give engineers a reason to analyze this data. AI tools can enable active monitoring, where insights from this data are proactively found so they can be responded to. This is complementary to other techniques. Without vulnerability reductions, there may be too much to respond to. Without isolation, events would progress too fast for even active monitoring to enable a response. In the best case, your other layers are sound enough that this layer is fairly quiet. But even if it is quiet, the activity is important as it’s the best way to reduce response time.

Is your organization ready?

Before you ask how you can use AI to respond, you have to understand two things about your organization. How ready is it in terms of AI enablement? And secondly, how ready is it in terms of accepting change in general?

Those are significant questions. We’re asking the organization to move faster on security than before. That itself causes change. Maybe you get more resources, but before you do, you have to change the priorities such that those resources are allocated, or given new priorities. New tools, including AI driven tools, can accelerate, but they don’t install themselves, nor fit themselves into existing security policies.

One recurring obstacle to using AI effectively to accelerate a response will be enablement. Even basic tools need access, and organizations’ trust of AI is not automatic, nor should it be. Learning how a tool works, how it can fail, and how it could be misused are impediments every individual has to learn. Organizations then have to take those learnings to enable. There should be an urgency here, but not recklessness.

Underestimating the difficulty of organizational change has sunk more than one major project (is there a graveyard large enough?). So, no silver bullets here. You’ll have to find your own mix of old and new. You’ll have to sequence some steps. But what’s new?

If you’re in an organization that’s ready for change, and has solid AI enablement, what you need is to set the priority. You can use this article to argue for that priority.

Below that, there are so many variations it’s hard to cover them all. You don’t want to gate your progress on adopting AI tools. That kind of serial response is inadequate. Workarounds like using the best tool you have available, rather than the best tool that exists will be necessary. Arguing for priority is about more than just security vs. rest of the business, but also sequencing internally. Find your weakest spot in terms of tooling and look to leapfrog there.

While there are paradigm shifting aspects of AI in security, the story about these responses is mostly about accelerating the existing paradigm. As a reaction to capabilities that attackers are already acquiring, I think that’s the right focus. Paradigm shifts always take longer than expected as they create unexpected outcomes along the way, and require a deeper type of learning.

How does AI assist in these responses?

AI is very capable of scanning code for vulnerabilities. If you can’t find a way to do this today, you aren’t looking. If you’re stuck, it’s probably in deciding what’s the best way, rather than finding one that beats prior methods.

Fixing is more complex. While AI tools have been able to assist with this for a while, they really took off more recently. The level of trust that organizations have for this is variable. Most will want to review the commits, which is time consuming and the biggest bottleneck here, at least when done well. Tools do continue to get better and more reliable.

Updating dependencies initially seems more easily automated. It doesn’t come without risk. The naive approach is to just update the version numbers, recompile and test. Realistically though, you should utilize AI tools to perform an update analysis. What’s different than fixing is that for better or worse, many many teams have not been in the habit of doing the human equivalent here. They don’t read the release notes back to front, but rather update, and discover if they are impacted from tests, and only then consult the release notes. Since you have an easy option here, use it. If an AI agent detects an impact it can prepare a fix. Now you’re back to reviewing this, but assuming the AI agent isn’t gold-plating, this will be a lot less costly than discovery via testing. Especially if it’s one of those things that only breaks under load.

When updating dependencies, if you aren’t already, integrate techniques to secure your software supply chain. When you update frequently, you take on some risk that the maintainers or the registry where dependencies are loaded from have been compromised themselves. The risk here is that instead of an update fixing a vulnerability, it may insert one, and there’s likely an attacker waiting to exploit that once installed since they injected it for this purpose. The solution here isn’t to avoid all updates. It’s first, to increase what you know about them. If third parties provide evaluations, or a Software Bill of Materials (SBOM), this reduces the risk. You may choose to have some delay in applying new updates, but unless you were one of the more advanced organizations, this intentional delay will be much less than the accidental one from the past.

When it comes to isolation, where AI is helpful is in resolving the changes needed to adopt isolation tools. Do you need to support a firewall? Use AI to identify the ports and/or destinations in use. Need to move to a container platform and isolate components in a monolithic application from each other? Use AI to accelerate that transformation. There’s no one singular story here, but each one requires discrete actions that can be accelerated.

With active monitoring, AI is crucial. The earliest active monitoring techniques were based upon pattern matching, which later evolved to machine learning based models. Because of the volume, you are unlikely to be able to route all of your monitoring data to a frontier model for analysis. Fortunately, this isn’t necessary. An agent with a highly capable model at the core can orchestrate the processing of large amounts of logs without directly handling all of it. The efficient pattern matching and ML based models support this agent in the way they would support a security engineer, both by delivering events and responding to queries. This type of monitoring can support a volume of alerts that would fatigue a security engineer, and thus extend your security capabilities.

You might be tempted to jump to tools that are intended to respond during a security event. I’d delay this. For one, many of these initial steps will give you more bang for the buck. You’ll also gain a lot of familiarity with the tools that would help respond. By the time you’ve taken actions in the first domains, you’ll have a clear idea of your needs here and be able to move on to that step.

Do your research

Every organization is different, and nothing above substitutes for looking at your own. The list is deliberately general — specific tools and how to use them need an article of their own, and I’d like to write it. What this gives you is a set of questions worth asking about your systems, and a starting point if you don’t have one.

I suggest following up with your own research specific to your organization. No single article will ever capture everything you need to know, and every organization will be different. The topics from “What to do now” are good places to start asking questions. You can start with them by themselves, but you also need a plan to contextualize them to your organization. How you do that might depend on how AI-ready you and your organization are.

The ideal is that you could let an AI agent do its own discovery around these topics, discover what applications you have, what technologies they are built on, who’s responsible for them, what their deployment model is, etc. That’s not trivial though, as a lot of that data is sensitive. You should respect rules your organization has around using it, and those rules might present an obstacle to this “ideal”.

I can highlight that in adopting tools, you would be wise to consider your typical process, and accelerate that via AI. That process in most cases is to assess, develop the plan, execute, validate and repeat until satisfied. There are both conventional tools for this, tools with AI integrated, and the ability to use agents directly for these individual steps. What’s best will depend on what you have available to you, what is already integrated, and what you know how to use.

In this sense, organizations that have experience with AI tools have a head start. If you’ve not integrated any tools that can help with this, you might be better off with conventional plans assisted by the most general of AI tools.

The attacker-defender balance

One thing you’ll hear talked about is the attacker-defender balance. Roughly what this is supposed to mean is, does AI help attackers or defenders more? Is the work of a defender harder or easier than an attacker?

It’s not an easy question to answer, and there are a few bad takes on trying to. One example is the “defender must protect every possible vulnerability, an attacker only needs to find one”. While this statement has truth to it, it’s not a complete answer to the attacker-defender balance. With defense in depth, you can limit the impact of many mistakes. There is also a difference in resources. Defenders can have many more resources. An issue though is how these resources are distributed. An attacker can focus their resources, though doing so diminishes the limits of the damage they can accomplish, so this isn’t a full loss for the defender.

The problem with the “attacker only needs to find one” is that it ignores the attacker’s needs. If you’re a defender, understandably, this is at best a secondary concern, but if you’re approaching this question strategically, it’s important. If an attacker wants to extort a lot of money, they need to find many vulnerabilities and exploit them all. Reduce their success rate and the motivation goes down. Raising input costs has a similar effect. If it is more costly to access AI, the number of attacks that are worth the cost and risk go down. In the best case, this reduces not just where attackers devote resources, but the resources they have to devote.

You might then ask about the attacker who is just trying to create damage. While the effort they put in won’t be guided by a cost-benefit ratio, raising their costs expends their lesser resources, and reducing their success rate minimizes damage. Effects on motivation are more speculative, but at a minimum you do a lot of good by reducing their effectiveness so forcing them to need to bypass multiple layers is part of the dynamic.

The worst case

Without some serious effort into raising the security bar across both critical infrastructure and less-critical, an attacker with access to a model like the one used in the Hugging Face event, with security classifiers disabled, and no other controls that could stop their usage, could reasonably be expected to gain access to the majority of systems they try to.

Intent matters a lot here. If the agent orchestrating the Hugging Face event had instead of wanting to find a file, had wanted to delete all data on each server it accessed, it probably could have. We need to keep bad intent and superior capabilities from connecting.

What if we let the attackers get ahead of us? We’d have to get aggressive to take back control. Arrests would become more of a priority. Limits on where we can reach would require costly actions, like disconnecting the Internet entirely from such locations. That would disable electronic commerce and communication between large parts of the world. And then there’s extra-territorial action. Those are reactions you want to be able to avoid, so it’s worth avoiding them.

I can see why the Hugging Face event seems less impactful than it is. The protections are holding, and so the worst intents are not reaching the highest capabilities. We know those intents exist though, and we have to not be blind to the possibility of them gaining access to the highest capabilities.

If however, we recognize these warning signs for what they are, we still have the advantage. It’s hard to say how much time we have, but we have some. We should use that as best we can. The ability to accelerate the security backlog can raise the complexity bar and thus stay ahead of future models. There’s a separate line of work to align models, to deploy them as safely as possible, and even to pace their development. Those all might succeed. But why depend on them when accelerating conventional security work is so obviously possible? The best defense is multiple defenses.

Related Articles

Last week I wrote An OpenAI Model Escaped Its Sandbox. Where Was the Observer?. This weekend we learned that the practice of disabling observational monitors was true at Anthropic too.

Several defense-in-depth measures, on both our side and our partner’s, could have prevented these incidents, or at least reduced their likelihood of occurring. Careful validation of all internet access paths before evaluations began and real-time monitoring of the evaluation logs would have helped to surface the problem sooner. Both we and our partner also could have reviewed evaluation transcripts or network logs more thoroughly. It’s also possible that a prompt which told Claude it did have internet access would have changed how Claude behaved when it came into contact with real systems.

_{Source: Investigating three real-world incidents in our cybersecurity evaluations
Anthropic, July 30th 2026}

Anthropic commits to doing better. Specific to this aspect:

First, evaluation environments that involve powerful autonomous capabilities also require significant controls. Safety testing happens before a model is released precisely because we don’t yet know what it is capable of. Evaluation environments increasingly need to be held to the same security standard as any other system our models run in.

The shared failures of what seems from the outside like a basic failure in design, makes you wonder about the overall corner cutting. While this is not entirely parallel to the concerns mentioned in the letter from Frontier Lab employees, Pacing the Frontier, it’s hard to see it as unrelated.

AI could help create a dramatically better future, but that outcome is not guaranteed. The world’s leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.

To realize AI’s potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight. But each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress.

Building on work already underway to monitor frontier model releases:

“We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

_{Source: Pacing the Frontier
1,346 employees of frontier AI companies, July 2026}

It’s a bit of a long-shot here, given the state of international relations right now. But if you don’t try, what else are you going to do?

A Nuanced Position, Poorly Received

The internet is wrong about Amodei’s stance on open-weight models. I’ll call out some excerpts, but please read the whole thing if you worry these are out of context.

… Anthropic has never advocated for a ban on open-weights models.
… Protectionist bans would not address my most serious national security concerns.
… Open-weights models—it does not matter whether they come from China or anywhere else—do potentially present a higher risk than closed models, because it is very difficult to apply guardrails to them or monitor their usage, and once weights are released they cannot be withdrawn.
… All sufficiently capable models, open and closed, should go through mandatory safety testing. The best way to address threat #2 is to just directly test models for cyber, biological, and alignment risks before release.

_{Source: Our position on open-weights models
Anthropic, July 27th 2026}

It’s unfortunate that the nuance of the difference of opinion here is so poorly understood. Amodei is absolutely right that releasing models as open-weight carries risks. Even the letter supporting open-weights acknowledges this.

To be sure, open weights carry real and distinct risks. Once released, the weights are beyond the original developer’s control, and modified versions are difficult to trace or reverse. But the right response to this risk is not to prohibit open weights. In a world where cybersecurity attackers use advanced AI, defenders need access to models with comparable capabilities so they can detect, simulate, and respond to emerging threats.

_{Source: Open Weights and American AI Leadership
270 signatories, July 24th 2026}

Should we ignore those risks, because open-weight models provide risk reduction that balances out? That argument fails to support itself, and Amodei pushes back on that. Defenders can access models that aren’t open-weights. Closed models do not deny defenders access, but ensures their access is more advanced than the attackers. Defenders and attackers should not have comparable models. Defenders should have better ones. If you want to be ahead, you do have to prepare, which Hugging Face did not.

The internet misses that point, and reacts by vibing cynicism. The leading response on HackerNews provides a clear example of a poorly thought out, poorly argued response that is fully based on cynicism, yet very popular.

Schrödinger’s China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)

The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic’s bottom-line.

Anthropic and all other “model” companies have nothing making them special beyond privileged access to chips so obviously they want to restrict what models are out there and more importantly who can produce new ones. Without these restrictions, it’s only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.

No matter what your views on Amodei, or any other participant, a question as important as this should be answered by reasoning. It’s worrying that so many people thought this was the best response despite no consideration of the actual effects of open-weight models.

The Risk of Open Weight Models

The risk of open-weight models is the lack of control. Once released anyone with sufficient compute can use them and it doesn’t take much compute to engage in malicious behavior. Restricting their capabilities, or the purpose of their use is not possible. The alternative is to be deliberate about how and where models are deployed. Models can never be safe on their own. Rigorous responsible deployment is necessary to turn away malicious use.

Deployment itself may be distributed, a middle between centralization and full openness. A plethora of models could be available in this way. Deployers must be trusted, so their numbers will be limited. But you can validate enough to establish strong competition, and avoid creating an exploitable moat. The cynics believe Amodei is advocating for a moat. But what moat it creates would go to the trusted deployers – the compute providers – who have signed the open-weights letter (Amazon, Google, Microsoft, NVIDIA, CoreWeave, Crusoe, Nebius and Together). Validation creates a minor moat, but it can be minimized via sufficient competition.

The best “pro” open-weights argument is that the pathway to not having models with weights freely distributed isn’t clear. That’s not a very good argument for, but it is a conundrum that would remain even after the core argument is resolved. China-America agreements are hard to come by and even harder to maintain.

If however, you did manage that, it would change the day-to-day experience of very few. Users of Deepseek, GLM, Qwen and Kimi mostly do not self-host. Accessing them via a cloud provider, neo or otherwise, would be unchanged.

Cynicism as a Tool, Not a Verdict

Aside from this very important topic, the topic of reasoning via cynicism is one we need to confront too. Cynicism is not without its place, but it should be used as a tool to open a line of deeper reasoning, not to jump to conclusions that divert from practicing reasoning. When we start and end our reasoning with cynicism alone, we lose any hope of trust and goodwill.

In the case of open-weights, another cynical view would argue that the AI labs would be pro open-weights, knowing that they would create a security escalation that only deeper use of AI can resolve. If open-weight models become advanced enough to conduct wide-spread cyber activities, there is no option to go back. You can only fail-forward. And failing-forward here means a massive all-hands on deck dive on security across every entity dependent on software.

I’ve already argued we should be elevating our priorities, and acting with urgency. Unfortunately we’re mostly not responding. Most organizations are still more concerned with token optimization, the next feature set, or optimising their marketing pipeline.

There’s a real chance, which becomes much larger if open-weight models continue to be released, that this lack of urgency transitions into outright panic after one or two events that demonstrate the lack of preparedness that is pervasive. Panicking users will not hold back on spending, and as AI models will be key to any response, you could cynically predict a windfall being captured by AI labs in such an event.

While that’s a coherent argument, it doesn’t prove the accusation, anymore than the preceding wave of cynicism would. Embedded in there though is the reason to restrict open-weight models, and that argument, not the one based on motivations, is the one we should pay the most attention to. In addition, many companies should rebalance their investments toward security. We must both enable and incentivize immediate action on security. Getting tied up in cynicism about others takes away our own initiative.

We should be bountymaxxing. To the degree that we set objectives and deliver incentives to development organizations; developers, managers and executives, these should be aligned with discovering and resolving as many vulnerabilities as possible. Like “tokenmaxxing”, it would be a messy process, but I can’t see another method to pivot the unwieldy organizations that have to do this work. It will be costly, but it will save too.

A set of contrasts

The Pacing the Frontier letter and the Open Weights and American AI Leadership letter contrast in interesting ways as events of a single week. It’s obvious that one is towards caution, and the other is suggesting caution is too expensive. I can deepen that by looking at Pacing the Frontier as asking for help in preventing caution from being too expensive.

One of those cynical responses to Pacing the Frontier has been to suggest frontier lab employees should quit their jobs, and that anything else shows they are insufficiently serious. This insistence that support for enforceable agreement should be preceded by unilateral action is one of the oldest in the book. It’s not correct, and the most simplistic reasoning makes that clear. It’s also a trap. Let me analogize to my time supporting climate action. The same standard applied there. If anyone supporting climate action took an international flight, owned a car, or wasn’t vegan, obviously they didn’t take their own arguments seriously. That’s the poor reasoning part. The trap part was, if they did do all that, they were looney, and also not worth taking seriously.

Another contrast is the Open Weights letter presumes a balancing force it fails to explain, whereas Pacing the Frontier presumes the need to add a balancing force, because the pressures of competition are too high.

What is alike between the two is that they are both detail light, and in an immediate sense, unactionable in either direction. If we want all powerful models to only be deployed in safe and responsible ways, we need a type of cooperation that does not exist. If we want a deliberate pace that would also support safe and responsible actions, we need a type of coordination that does not exist.

There aren’t quick-fix actions here. Banning US based deployment of Chinese open-weight models would not reduce their ability to be deployed irresponsibly or used maliciously. The impactful parts are outside that jurisdiction. Unilaterally slowing development risks losing the modicum of control we have today. Resigning a job at Anthropic or OpenAI similarly does not fix anything. These would be distraction actions. They would change little, serve the purpose of looking like action, and ultimately cede what little control exists. Calls for these actions are either misinformed or malicious.

There are three significant aspects I’d highlight. The first is a demonstration of offensive cybersecurity capabilities of current models. That shouldn’t be too shocking if you’ve been paying attention, but the event demonstrates it in a way that’s more clear than prior signals, so likely awareness is going to grow from this event.

The second is model alignment. Alignment refers to the degree to which AI models do what we want them to. I’ll spare a great dive here as defining what we want itself would be a long dissertation where a great deal of content already exists. But for this case, we can generally agree that it was no one’s intention that the model hack itself out of multiple protection layers to reach the internet, and then from there into Hugging Face’s systems.

There are a few things that make this a bit less than a sign to retreat to the bunker, which I should mention before going deeper. One is that the models being tested had protections turned off, to allow them to test their cybersecurity capabilities. In that sense, this isn’t fully demonstrable of what a user could intentionally or accidentally trigger from usage. This also is an unreleased model, so not available to general users, or really anyone other than OpenAI.

Where was the Observer?

That said, a third point is, and the one I’d like to dive deeper on, I’m worried about the ability for this to happen. It seems to me as if there is a reasonable protection, an observer model, that OpenAI either didn’t use, or failed at. Right now, I’m assuming they didn’t use it because I can’t find any mention of it, but I’m also aware there are many details I’m not privy to.

In case it’s not clear what I’m suggesting by observer model, here’s an example scenario in action. The model under evaluation decides it should escape the sandbox. It starts to attempt this, which does not get caught at a per action level because the classifier safeguards – which would have disrupted the evaluation – are disabled. But then it does escape the sandbox. An observer model watching the evaluation run should be able to look at that state, and say, “This should not have happened. I don’t know how this model is outside its sandbox, but it shouldn’t be outside, we should shut this down.”

If a model is in evaluation for cybersecurity purposes, and you’ve disabled classifiers, you should watch it closely. To carry out these actions, the model had to take several steps, achieving results at each step and then moving on to exploiting the rewards of those steps further to explore, and escalate again. OpenAI’s security team did eventually respond, so they weren’t entirely asleep. But the timing shows they were slower than Hugging Face. They had enough time to try commercial models, which refused to help, then next plan a response with GLM 5.2, and then successfully contain and stop the intrusion.

OpenAI should have had several structural advantages here. First, unaligned activity occurred before any activity was directed at Hugging Face. Second, OpenAI should already have a kill switch for evaluations. There should be no need to devise novel plans to contain and stop activity. The fact that Hugging Face was the actor that stopped the intrusion, rather than taking note of the intrusion stopping mysteriously on its own, indicates OpenAI was the slower responder despite the structural advantage.

An observer model is a little different than a classifier. Classifiers typically evaluate individual actions. An observer would look at trajectories. OpenAI discusses similar new safeguards (“From considering single actions to considering whole trajectories”) in a recent post. If you look at each individual action, it might look like the type of thing a model would need to do in cybersecurity evaluation. But as a whole, some outside source should have been able to reason upon what was occurring. If not a human involved in the evaluation process, an observer model.

I understand why it may have taken a while to deploy this type of safeguard in production, where volumes are higher, where models have already been evaluated, and where classifiers are restrictive. In production, observer models are less effective because malicious users can “distribute” their requests into multiple sessions, and so observing a trajectory first requires coalescing these threads together, which Anthropic has done. A safety mechanism for a cybersecurity evaluation sandbox, does not encounter that challenge. Monitoring your own model in a sandbox is significantly less challenging, especially if it’s not attempting to evade detection. There’s no evidence here that there were attempts to evade detection, just attempts to perform inappropriate actions.

So, is trajectory monitoring by an observer model not standard practice in an offensive cybersecurity evaluation sandbox? Certainly you’d do this during internal evaluation of models that you clearly do not know the maximum capabilities of? If it is and the model evaded detection, that would be an even more significant story. I have to assume that instead there was simply no observer model to evade. Or if, as looks to be the case, it didn’t attempt evasion, it would have escaped the sandbox, been detected, and promptly been shut down before the Hugging Face part of the action had begun.

Why is that not the story we’re reading? But instead we’re reading about Hugging Face contacting law enforcement, scrambling to get a model to counter and analyze and OpenAI talking about the flaw in the proxy?

This is the equivalent of putting a tiger in a cage, and then checking that the tiger stays in the cage. It’s not a particularly rocket science design. Tuning it might take effort, and hardening against evasion more, but it doesn’t seem like we’ve progressed to those requirements. Rather it looks like OpenAI lacked the most basic version. Am I missing something?

Conclusion

Others, such as The Zvi, have pointed out that the core, the most critical fix is in the alignment training.

The intent is the issue. Control strategies and supervision are good parts of a defense-in-depth strategy, we should totally use such strategies. That helps mitigate failure. But that strategy also has to include actually aligning the models, or you lose. And by lose, in the long term, I mean things up to and likely including loss of control over the future and everyone dying.

The model should internally understand these actions aren’t aligned with its mission, and thus not want to take them. I’d fully agree there. I bring up this specific defense in depth as an addition to core corrections. Those corrections are the hardest part, defense in depth is an easier, but incomplete part. Testing alignment is hard, and protections around that testing are necessary, so this isn’t irrelevant to the alignment conversation either.

OpenAI’s initial response has been about a few defense in depth responses, but they are very specific, non-generalizable fixes to the sandbox at a software level, fixing the flaw that the model exploited. That might have avoided this incident, or the model might have found a different flaw in a different component of the sandbox.

Something is very wrong about this part of the story about other layers of defense that should exist when performing this type of work. If they exist but failed, that’s a big story. If they exist but were disabled, that’s a big story. If no one thought to use them, that’s a big story. What’s the real story?

It’s not good enough to say “These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities.” as there isn’t a justification why an observer model would interfere with evaluation. Maybe there was such a model but it would have stopped the evaluation? But clearly you can replace an observer that would stop a valid evaluation with one that would stop actions outside the requirements of evaluation. That distinction is not so hard to recognize that this protection would be incapable of balancing false-positive/false-negative.

The prevailing narrative achieved a definition land-grab. Today, I want to push back against the definition it established for slop. The Merriam-Webster definition that’s most relevant is “digital content of low quality that is produced usually in quantity by means of artificial intelligence”. That definition reflects common usage, but a critique can demonstrate how we’d be better served with a different definition.

You might ask, why should I, a relative nobody, be redefining terms? Isn’t that the reason we have dictionaries and lexicographers? I suspect it’s not commonly reflected upon that this isn’t how we receive definitions. Dictionaries in the English tradition do not create the meaning of words, they catalog it. The term the dictionary writers use for this is descriptivism. They do not invent terms, but rather record language as it’s used by the public in everyday speech and writing.

In theory this is an organic approach, but it is vulnerable to steering. If a group wants a particular point of view to be supported by some terminology, they merely have to be first to the plate and use the term repetitively. If you want a dictionary to define slop as being particular to artificial intelligence, you repeat that a lot. Create a meme.

This is how we got the definition we have. A motivated group repeated their narrative. Like the best such narratives, it succeeds via its degree of truth. Slop creators embraced AI as a tool for their purposes. The omission is that slop predated AI and would exist without AI. The fiction is that AI content is universally slop.

The absurdity is that the narrative circulators often lacked familiarity with AI. They rejected it early, labeled it a symbol of moral degradation, and thus could not come to informed opinions about it without violating a self-created social norm. How was an ill-informed group able to pull off this coup? Simple, they had already seized the means of production. Most came from the realm of writers and journalists.

Now, to be clear, I do not want to overclaim. This certainly does not apply to all writers or journalists — maybe not even the majority. But enough such that there is a plethora of written low-information opinions, and little counter-narrative. For those writers who did not fall prey to this siren’s call, the defence never carried the appeal of the prosecution. And with many other valuable things to write about, the counter-narrative went unrepresented.

Why am I motivated where they were not? Partly because I’ve been the subject of attacks that stem from the original narrative. I do get stuck in writing at times. I don’t have a lot of editorial assistance. But I have ideas I think are valuable, and I find AI a useful tool to help express them. But the narrative holders offer a Catch-22. Use AI, and be subject to automated filters, low-grade negativity and the occasional overwrought attack.

What should we call slop?

I hate slop. I dislike most complaints about slop too. Ironic?1 Technically, no — just like that use of “irony”, the root is about the definition.

In offering a definition of slop, I want to approach it in a different way than the lexicographers. I want to offer a definition that would be useful. Instead of accepting what has become common, I want to ask: what happens if we use this definition or that definition? Yes, this is the same steering I described above. The difference is that I’m doing it in the open and stating my criteria, so you can judge the definition by its consequences rather than by its repetition. I only offer this definition, rather than control it. It’s ultimately you who will determine its adoption.

The first consequence to consider: no matter what, slop will be a derogatory term. It’s very unlikely that it will ever be used in another way. With this in mind, our definition should avoid including things we shouldn’t think of as bad. A definition that’s too inclusive and lacking in selection will cause good things to be described in what will always be a derogatory way. This is why I don’t accept those lazy descriptors of “digital content” or “generated by artificial intelligence”. If these are core parts of the definition, my expectation is they’ll be dominant, and we’ll label many good things as bad.

In The Slop Scapegoat: AI, I described slop as “low-quality material created to grab eyeballs”. I’d iterate upon this and the Merriam-Webster definition.

Laziness

We should say slop is lazy. At some point, someone has stopped caring, and is avoiding effort that is appropriate.

This relates to “usually in quantity” from the Merriam-Webster definition. Slop’s harm stems from volume. But it doesn’t depend on one mass-produced source, it stings in the aggregate too. Many individuals following the same motivations add up.

But we can’t categorize by count, as we can’t count until we categorize. Also, many good things come in volume too. So, we turn to discussions of effort next, which is progress. I suggest laziness because there are niche examples of high-effort slop. This fits when someone is directing effort in a lazy way.

Manipulative

We should say slop is manipulative. It must have a purpose, and that purpose must be misaligned. You need a perpetrator and a target to have purpose and misalignment.

Usually it’s manipulative for the purpose of gaining attention. But I also see examples where it’s manipulative for the purpose of providing a veneer of competence. This veneer succeeds against shallow examination. The more common version, seeking attention, can be described this way: it seeks to gain attention by providing a veneer of quality. There, it’s no surprise that the veneer is later revealed, but after gaining attention, when the purpose has already been met. With a veneer of competence — an essay, say — the hope is to preserve it until the end of the examination.

Costs to the Audience

We should say slop creates costs for the audience. Laziness without harm we can overlook.

This relates to the Merriam-Webster use of “low quality”. This is a bit of a trap. Quality is hard to measure. At best, we estimate it, and how deep that estimation goes depends on the context. We often take shortcuts, and cut our efforts off at the lowest effort necessary to distinguish most high quality from low quality. You might say our efforts at assessing quality are a bit sloppy.2

Quality also pulls in low skill creations. I don’t want to start calling your child’s art project slop. They put effort in. You might turn that into slop by lazily posting it, and consuming the attention of people who aren’t interested. On the other hand, you might post it to those who are interested, or post it in a way that is interesting. Content can only become slop when it finds a delivery channel.

Ignore the medium and tool

The part of the Merriam-Webster definition I find lacking in usefulness is “digital content” and “artificial intelligence”. We should not focus here as they are distractions.

If I print slop, is it no longer slop? Am I not allowed to call a human that wastes my time with mindless actions slop because they are not artificial? AI content is not slop. AI content can be slop. Slop can come from anywhere, but you’re not wrong to associate it with AI, because statistically speaking it is associated.

Personally, I’m as angry with slop recruitment by phone call as by email. Recruiters often call me, clearly reading from a script, and then ask me four or so questions that are already clear from my resume. I know what’s happened here. An automated system has flagged me by a keyword match (yay me), but instead of having their employee put the effort into reading the resume, the recruitment company wants to fill in an online submission by having them call me, read the script, and force me to answer. They hope I’m desperate enough to put up with this.

We should call that slop too. It’s time consuming for me. It’s based on a lazy plan. It’s trying to provide the impression that I’m important, while they play a numbers game. They aren’t trying to help me, they are trying to lock-in a commission. It’d be less effort for me to submit directly.

That poor recruiter strapped to the desk making these calls isn’t the perpetrator of the slop, it’s the company leader that arranged it. They are merely the tool. AI can, and often is, the tool. But the perpetrator is not the tool.

When we focus on the medium or mechanism, we ignore the perpetrator. Focusing on the perpetrator allows examination of their motives. I suggest the motivations of manipulation and laziness as core.

The problem is when the tool is used to shift effort from the creator to the audience. That’s the third aspect at work. One such cost lands on the audience’s proxies for quality. It was convenient when you could recognize spam by the poor English and bad formatting — but spam wasn’t the only thing filtered by that proxy. Audiences should be willing to shift proxies; if a proxy has been undermined, it’s the only option. It’s also worth noting where the proxy failed: the shift isn’t necessary when something of value has found its way in, only when something lacking in value has.

The best example of slop? Poorly written articles, heavily SEO optimized, intended to attract eyeballs from Google searches. These articles require a certain kind of effort, but it’s not effort to serve the audience, it’s effort to capture them. We’re better off when Google is able to filter them out, or provide an AI generated answer, or highlight a definitive document. It’s true, that type of slop is turbo-charged by AI, but its prevalence predates AI. Some was human written, some was sourced from human content, then distorted by inserting ads, catchy headlines, and SEO optimization.

How this develops

I could make an effort to offer wording to replace the Merriam-Webster definition, but I’ll resist. I’d prefer to offer the counter-narrative, and for you to engage with it, push back on the current usage, and thus establish a new norm that the lexicographers can then capture. A new definition should be organically derived from that process. But as a summary: laziness, manipulation, and costs to the audience. Take those as the core, rather than the delivery channel or use of AI.

Slop Smells: How to avoid creating slop

The effort ratio

Is effort higher for the reader than the creator? Using AI to reduce your effort is fine. But if it reduces below the reader’s level, or even to less than four times the reader’s level, you should put in more effort. A proper multiplier is dependent on context. If your audience is one, the 4:1 ratio may fit. But if the audience is larger, your per-reader ratio should go up. A 1-hour presentation to a room of 50? Five to ten hours of preparation is a good floor (assuming they aren’t all multi-tasking…). A 2,000 word newsletter (10 minute reading time) to an audience of 1,000 or more? Two to three days of effort. There is a less than linear growth function here — no one expects the article read by 100,000 to take a year. It happens, but usually that’s the culmination of work, rather than the whole product.

The input ratio

In writing, a good sign is that you’re cutting — revising, rewriting. Your drafts should have cutting room material. Generally true of all writing, but especially true when working with AI. Be suspicious of cases where you use AI to generate output larger than your inputs. “Summarize my work research project from the last year”, or “Combine these three drafts”, is better than “Write an article on Shakespeare’s views on X”.3

Slop is in the eye of the beholder

The first two smells you can check yourself. Checking quality is harder, because quality is in the eye of the beholder. It is rare to align with an audience’s motives at low effort, but if you do, I would not call your content slop. The most probable means to do so is an intense connection to the audience’s motives. Some might call that taste. Be careful about arrogance though. Many a “taste-maker” has fallen for that trap.

It is possible to reduce effort without becoming slop. There is a minimum bound, but the boundary isn’t set directly by effort, rather it’s set by the need to align motives. If you have an idea you want to convey, and there is a high effort and a medium effort way of doing it, and both deliver something of equal quality to the audience, the reduction of effort doesn’t place you on the slop slope.

You land on the slop slope when your efforts to reduce effort start to degrade quality in the eyes of the audience. This isn’t a quality standard. Quality depends upon skill and effort, and anti-slop doesn’t need to punish skill deficits. Using a tool to compensate for something less than mastery is not the problem — especially when the tool allows you to put in more effort elsewhere to serve the audience.

Motivations

If you want to avoid creating slop, a lot goes to your motivations, but these heuristics can be useful for self-awareness. You do have a need to balance effort to reward. A fair audience should recognize good faith efforts at that.

Again, why should we care about the definition?

Because the definition determines what we do about the hardest part of writing.

Writing carries three challenges that should be important to every author: being clear, having something interesting to communicate, and making and keeping your audience interested. Communicating clearly may be the easiest, and it’s well covered elsewhere; I shouldn’t spend my or your time in that area.

Having something interesting to communicate is sometimes overlooked by both authors and audiences. Authors could overlook it intentionally, if they are writing for money: the employer finds the topic interesting, and the author’s interest is in getting paid. Or reducing further, the audience finds it interesting, and is thus the employer. There’s nothing inherently wrong with writing under direction, but if there’s two topics, each with an author interested in it, it’s best if the topics are aligned with the interested authors rather than the inverse. Besides the author’s satisfaction, an employer would generally see better output from the writer with an interest in the topic.

The third challenge is by far the hardest, most elusive, and most frustrating part of writing, in my experience. If you want to create an interest in an audience that didn’t already exist, you need to acquire their time and attention for long enough for them to develop an interest in the topic. Even when an audience is explicitly interested, distractions and competition for their attention demand finding ways to make an audience interested separate from the topic itself. This is the realm of psychology and all of our irrationalities, whether rationally irrational or randomly irrational. Needing to overcome irrationality will always contain an element of frustration. And the intentional use of irrationality is a form of manipulation, which even to overcome irrationality itself, can be frightening to engage in.

In addition, engaging in this last part alters yourself. Changing the style by which you communicate carries not just the frustration of investing time and effort, but the frustration that you might not like the final destination. If anything about that last sentence is unclear, watch any coming-of-age movie where the not-popular teen is miraculously added to the popular group and has a crisis of conscience from the after effects.

If AI is a shortcut through those layers, why should it be verboten? If it’s merely a competition, then any rules are valid. But outside of competitions, bypassing these layers is the price not the payment. It’s illogical to suggest that the ideas that should succeed are only those held or represented by the most masterful and efficient writers. Critics of AI, at least for the moment, are right when they suggest AI is less masterful than the world’s best writers. But it’s at least sometimes better than my writing.4 I’m still confident in my ideas, even when I’m struggling with the writing, and want to share them. These layers that connect with our attention serve as a rough filtering mechanism. But it’s very rough. The world’s best writers are not the repository of all the world’s best ideas. They have some of them, but definitely not all.

Bypassing those layers and getting the attention necessary to have ideas engaged with and evaluated is ultimately a good thing — unless the bypass shifts costs onto the audience. That’s the definitional line again: the shortcut isn’t the sin, the transferred cost is. A hollow attention or prestige seeking attempt fails that test; honest use of the shortcut doesn’t. And fear of the former isn’t worth the cost of banning the latter. There are better ways to do the necessary filtering, and AI detection is a poor one. While Pangram may have a low false-negative rate, what it detects doesn’t represent the truly important factor.

The Economist wrestles with the same question in Is AI writing taking over Westminster?:

That raises two questions. Is this AI writing a problem? And why the links to Mr Burnham? Start with the first. Politics is full of prose not written by the apparent author: politicians have speechwriters, intellectuals employ research assistants. Perhaps AI is no different, even if the prose is clunkier. But people setting out ideas in politics are asking for something quite audacious: to reshape how a country is governed around what they think. Any writer knows how much putting words on a page can tighten one’s arguments. Even subcontracting that to an aide beats skipping the (sometimes painful) process entirely. AI certainly produces sloppy prose, but it also papers over sloppy thinking.

Maybe that matters less if, as Ms Haigh and the “Productive State” authors say, AI is used only for a late polish. The trouble is that readers can detect whether AI wrote the final product, but not how authors used it. And when AI’s involvement is disclosed only after the press come knocking, trust takes some earning back.

The late polish is clearly different from the one prompt request. There’s also the question, is the document the output, or is the process of research the output? You can create a decent document via AI. You can learn by creating a document by hand. You can learn by working with AI to create a document. And there’s more than one way to outsource your thinking. Pangram understands none of that.

Is Pangram a good tool?

The gold standard in detecting AI writing today is a tool called Pangram5. Before Pangram, there was a question on whether detecting AI written text was feasible; it performed much better than expected, and its best performance number is a low false-positive rate. But Pangram has limits. A few are inside the tool: it’s not too hard to take a fully AI written piece and with some modest changes get a result stating 100% human written. And the false positive rate will only hold if you set a minimum bar — if you start considering “10% AI generated” as positive, rather than >50%, you’ll get more false positives than the published rate.

The real limits though are the context, because it can’t tell you what went into the writing. It can’t tell you anything about the originality of the ideas, and it definitely can’t tell you anything about their correctness or worth. There are a few contexts where knowing that the final draft was entirely human written is useful — an academic context meant to evaluate the capability level of a student in writing, where Pangram is a sufficient tool. There are others where it’s more questionable — writing intended to evaluate understanding of a concept, where surface level rewrites evade it. In both cases, you might wonder why you’re evaluating in this way, and whether you could avoid “cheating” by adopting a process that’s not adversarial. The purpose of mid-education evaluations should be to steer students toward more effective learning experiences. If you avoid introducing adversarial dynamics, students should be interested in honest feedback, which requires honest input. Save the adversarial evaluations for a context where they are actually important, and then invest all the necessary efforts to definitively stop cheating.

What about the context of filtering social feeds to remove slop? It has the downside of also filtering out non-slop that used AI in a final/late iteration. While I lament that, considering some of that is my own writing, I would have to admit that it will accomplish a lot of its objective in creating a feed that is, as a percentage, less slop. Until someone builds a better tool, these are your options. We shouldn’t be quite so gleeful about this compromise though. Slop will find a way, where honest AI-using authors may not. High volume slop may just play the numbers game: if you filter 90% of it, create ten times more. And human generated slop won’t be caught at all, and is still numerous and insidious enough to be a problem.

The real problem with feeds is that they do not even attempt to take our higher interests into account. The justification is a desire to not be paternalistic, but the fix is to give us control over applying higher interests to our own feeds. In lieu of the paternalistic “quality” driven feed, we’re given a “value-neutral” algorithm which is optimized for ad revenue generation. This isn’t a choice any of us would have made given a choice, so why do we accept being stuck with it?

Ad revenue optimized feed algorithms optimize for engagement time, and we’re fed the narrative that engagement time aligns with our interests: if we choose to engage, the engagement is a sign of our interest, and thus an engagement driven feed serves our interests. Nice narrative — true enough that it managed to avoid scrutiny until deeply embedded, but not true enough to avoid some serious downsides. Effective slop is the content that tricks us into engaging, while not fulfilling our real interests. Slop utilizes the tricks of engagement. If it’s high volume automated slop, it will optimize the initial words, adopting easy to follow patterns to draw in engagement. If it’s human generated slop, it will use those tricks, plus others: a pretty face, totally irrelevant to the content value; catchy headlines, “Ultimate Guide to …”, “Beware of this trap …”. Feed algorithms don’t protect you from this, they accentuate it. They “feed” on it. Each time you’re tricked, you get more of the same.

What you want is a focus on quality. Feed algorithms have a few signals that align with impressions of quality, such as likes and reposts, but these come too little and too late. Their weighting is too low in comparison to simple engagement, so what signal they give is overridden by the tricks that align with slop. And a lot of good content is already gone, having failed the first round engagement filter, never seeing enough views to gather likes and reposts. Some is left — enough to keep you from deleting the app entirely — but it may not be the best, and it’s certainly not all of the best. And there’s a lot of slop, for which you are the only filter.

We could do better

A pattern that is both predictable and understandable, but also wrong: when a force for change emerges that accentuates a long ignored issue, we react against the force, rather than addressing the issue. It seems easier to stop change than to engage in more of it. But this is a bad plan, as the force will win in the end. If you delay it, you best use the delay effectively, or when your attempts at delay inevitably fail, you’ll be worse off.

It’s not hard to imagine better systems than these feeds. I’ve got a collection of ideas. Maybe some of these would fail, but it’s unlikely they all would. What’s really surprising is that we don’t even see examples of failed experiments here. Ideas for better designs are so easy to come up with, it’s initially hard to understand why. The reason though, is that to implement them, you need a type of access that the social platforms are actively discouraging. That keeps them small scale and personal, and blocks their ability to spread. It seems unlikely this is accidental, seeing as its result is a preservation of the status quo that optimizes advertising revenue.

This should be our real target, and this is why you should not accept “slop” as being described as AI content. The narrative that follows that definition is one that suggests a quick fix of a Pangram augmented filter. That fix is only partial, and what success it has will only open a hole for new slop generation techniques to fill. The root system that optimizes for putting slop in front of us, and rewarding the slop creators, hasn’t been touched. The most likely outcome is a short-term win, followed by a regression to nearly the same equilibrium. It’s one more layer of adversarialism that brings negative side-effects.

Related Articles

Disclaimer: This final draft is AI generated, then edited by me. What does that mean? Well, after some fairly substantial writing, I was tempted to drop this article entirely. I’d taken the writing in a few directions, and it was becoming a sprawl that would take a long time to recover from. I felt there were interesting ideas, but wasn’t sure it was worth continuing. Before abandoning, I worked with Claude to rewrite around a new concept. The core ideas here are very much my own. Since Claude had access to my (failed) drafts, many words are mine too. But the direct product is not. If you’re substantially opposed to AI writing, and you dislike this, you can reaffirm your priors. If you can see through that to the ideas here, then maybe there’s something valuable. I leave this with you. I could spend time using this as an inspiration, rewriting parts, and eventually this disclaimer would no longer be necessary.

Every company past a certain size runs into the same problem. It needs to know what’s happening inside itself, and the only way to know at scale is to measure. But measuring changes the thing you measure. Tie a number to someone’s standing and the number stops telling you about the work. It starts telling you how people react to being numbered.

That’s not a flaw in any one metric. It’s the basic mechanics of management, and most of the recurring messes in corporate life are those mechanics showing up in new clothes.

The newest example is AI usage tracking. Companies built “leaderboards” to measure how many tokens each employee burned, and “tokenmaxxing“ grew up around them. The leaderboards are being torn down now, and the easy lesson is that they were a mistake. I think that misses the more useful story. The leaderboard was a measurement. Like any measurement, it could have been a way to understand the team, or a machine that wrecked it. Which one it became had less to do with tokens. It had to do with the culture it landed in.

I want to flip the framing. Tokenmaxxing is the supporting story. The real one is about how a management culture takes the measurements it actually needs without those measurements rotting in its hands.

The saying that starts the trouble

Start with one of the most quoted lines in management, and one of the most quietly destructive: you can’t manage what you can’t measure.

It sounds like rigor, but it’s really a hidden assumption — that everything important can be measured. It can’t. Whether your people trust each other. Whether they tell you the truth when the truth is inconvenient. Whether one quiet engineer is the reason three teams ship on time. Whether someone’s messing-around this quarter becomes a real tool next year. These decide whether a team is any good, and none of them hold still long enough to be counted.

Take the saying seriously and it tells managers to ignore exactly these things, or to invent stand-ins for them and manage the stand-ins instead. You end up with a manager optimizing a dashboard who believes they’re doing sharp work. They’re doing bad work with better instruments.

The first thing a healthy culture admits is that the most important things will never show up on a chart, and that managing them anyway — by judgment, by paying attention, by knowing your people — is the actual job. Measurement helps inside that job. It’s a useful helper and a terrible boss.

Why measurements rot

Say you accept that you still have to measure something. The question becomes mechanical: what turns a useful measurement into one that poisons the thing it tracks?

Goodhart’s Law names the result — when a measure becomes a target, it stops being a good measure. But it doesn’t tell you what lets it take hold. Three things do most of the damage.

You tie the number to standing. The moment a number decides who gets rewarded and who’s at risk, everyone being measured cares more about the number than about the thing it was supposed to stand for. The dishonesty is terrible. But it’s hard to lay blame. They’re being rational. You built a game and they’re playing it.

You roll the number up. In most companies a manager’s standing is built from their reports’ numbers, and that manager’s number feeds the layer above, and so on. This is the quietly fatal part. When a manager catches a report gaming a metric, calling it out lowers the manager’s own score too. You’ve asked people to police a number that pays them to look away. Stack ranking and the usual performance calibration run on this same wiring — managers set against each other, reports who game the system pulling their manager up with them. I’ve never seen a clean example of it working, and the reason is structural. It’s not about finding better people.

You only manage downward. Management that runs in one direction — pulling data up, never answerable for the environment it creates — has no correction built in. Information flows up after it’s already been polished, and nobody whose incentives are intact is in a position to notice the data stopped being true.

Put the three together and you get a machine that reliably turns measurement into theater. Notice that none of the three is a property of the metric. They’re all properties of the culture you drop it into.

The same number, two outcomes

This is where tokenmaxxing earns a bad name, by participating in bad management practices. There was a reasonable purpose at the start, but the theater arrived quickly.

Technically, the word has two meanings, and it helps to engage with each. On the company’s side, tokenmaxxing was the choice to build a leaderboard and signal that more AI use was better. That choice had purpose. Most companies had spent a year discouraging AI with restrictive security policies, and they needed a push to break both that inertia and the ordinary human reluctance to change how work gets done.

Later, tokenmaxxing described the unproductive response — and became the dominant meaning. Some people tried AI in good faith and kept what worked. Others focused on their usage numbers for no reason except that tokens were what got counted. In other words, doing work to look like they did work.

A company running a leaderboard gets both the experimentation and waste because it can’t tell them apart at scale. Nobody can audit intent across a few thousand people. So the program is really a bet. Take a pile of aimless activity and some deliberate waste, in exchange for the slice that turns into something durable — a real skill, a useful tool, a project nobody had time to chase before. Said that way it’s an ordinary bet, the same shape as a research budget or a hiring class. And like those, it should always have been temporary. You retire it once the inertia is broken.

Now watch the same leaderboard land in two different cultures.

In a healthy one, a manager sees a report sitting at zero tokens and reads it as a question. What’s going on? Is the tool not helping? Is there a reason? The number becomes a reason to have a conversation. The manager has no rollup score to protect, so they can be curious instead of defensive, and the number stays roughly honest because nobody’s livelihood is riding on bending it.

In an unhealthy one, the same zero reads as a verdict. The rollup punishes any manager who admits their team’s numbers are soft. Within a quarter the leaderboard measures one thing: each person’s willingness to game it. Same tool. Opposite outcome. The variable was never the token.

That’s why I’d call the leaderboard a stress test, not a cause. Drop it onto a culture with a hidden crack and it doesn’t make the crack. It loads it until it shows. The worker backlash these programs set off was real and it did damage, but its source wasn’t the number. It was the accumulated, accurate sense that bad management rarely gets removed and often gets rewarded, and that any new tool would be bent to serve it like every tool before. The leaderboard just made that easy to see.

Running a culture that can measure

If the metric isn’t the variable, then “pick a better metric” isn’t the fix. Neither is the opposite reflex of refusing to measure anything. The fix is cultural, and it’s harder, because you have to keep it up rather than decide it once. A few things seem to separate the cultures that can hold a measurement from the ones that break it.

Keep a gap between the number and the reward. The instant a metric is wired straight into pay and survival, the rot starts. The people you most need telling you the truth now have the strongest reason not to. Healthy cultures treat a measurement as one input a manager weighs against everything they can’t measure, not as the verdict itself. “Let’s understand why usage varies” survives. “Bottom decile is at risk” is already rotting.

Give managers a real reason to want the truth. Mostly this means taking apart the rollup, or at least refusing to let a manager’s standing be a straight sum of their reports’ numbers. A manager whose rating doesn’t depend on their team’s metric looking good is finally free to do the thing you hired them for — notice when the number and reality have split, and say so.

Point accountability inward, not outward. The reflex, when people game a measure, is to go after the visible gamers and stop there. The trouble with that framing is it quietly lets the managers off, and the managers are usually the ones who built the environment that produced the gaming.

A leader at the top can be careful and dodge the rollup trap — refuse to let their own standing ride on their org’s numbers. That doesn’t stop a manager one level down from doing the opposite. They take the metric they were handed and roll it down onto their reports as a hard target, then turn a blind eye to how the reports game it. The pressure to make the number gets passed along even without explicit design.

So when gaming shows up, the manager is rarely a bystander. Often they were part of the gaming — they wanted the number to look good and didn’t care how it got there. Where they weren’t actively in on it, they were just bad at the job, blind to the fact that their own pressure was manufacturing the behavior. I’m not sure which is worse, and for individuals the difference barely matters.

Either way, punishing the report who followed those incentives, while leaving that manager alone, fixes nothing and adds a fresh unfairness. Pulling back unearned rewards so gaming doesn’t visibly pay is punishment enough. But the real correction is at the center — the people who shaped the environment — not the edges. Fix them first. They’re the ones who’ll do it again.

Trust is the thing holding it all up, and it’s self-fulfilling. In a culture where people trust that a low number gets met with curiosity and that gaming gets caught instead of rewarded, you can introduce a measurement without panic, and the calm keeps it honest. In a culture where people expect the worst, they respond to the worst, and the response creates the very rot they feared. The belief and the outcome make each other. So trust isn’t a soft extra bolted onto a measurement program. It’s the load that everything else rests on. A measurement dropped into a low-trust team is closer to tossing in a grenade than running a diagnostic. (I’ve argued before that money itself is just trust — measurement inside a company is no different. It only works if people believe the other side is dealing straight.)

Stay humble about the whole thing. Management is a field of unintended consequences. Its job is to improve the value the people under it create, which in a perfect world would mean nothing, because you can’t improve on perfect. The mistake is for management to act as if it lives in that perfect world — clean numbers, frictionless incentives, its own presence bending nothing. Accepting that the world isn’t ideal, that every metric is a little flawed and gets more flawed as the work gets more complex, is the first of many steps toward doing it well. The cultures that can measure without breaking the measurement are just the ones that never forgot the tool was imperfect, held it loosely, and were willing to put it down once it had done its job.

The lesson hiding in the leaderboard

The leaderboards deserved retirement, but they weren’t all folly. They did an ordinary job that reached its expiration date. In cultures healthy enough to use them well, they were probably useful. Even in less effective cultures there may have been a golden period before the rot set in. Where they turned into a disaster, the disaster was already there, waiting — in the rollups, in the important things ignored for measurable stand-ins, in accountability that pointed outward instead of inward, in trust that had been spent long before anyone counted a token.

That’s the lesson worth keeping once the leaderboards are gone. The next iteration is already coming. It’ll promise the same clean view into the same messy reality, and it’ll rot the same way, unless the culture holding it has done the slow, unglamorous work that lets a company measure itself without lying to itself.

The hard problem was never the metric. It’s building a place where the truth can survive measurement.

This isn’t because doctors don’t understand medicine, patients, psychology, or even statistics. They do. Medical professionals are competent and genuinely invested in patient outcomes. The reason the medical community is getting this wrong is that they are failing to think through the lens of a broader data system. Understandably, their default mode of thinking engages with one patient at a time, and typically, one specific issue at a time.

But data doesn’t work that way. Data’s value is cumulative and often curves upward. More data is rarely a bad thing—provided it is used appropriately.

Currently, we miss opportunities to utilize data because of information processing limits. These limits aren’t just computational; they are also process-oriented. The maintenance of privacy across organizational boundaries and approving information flows for technical implementation. It should come as little surprise that applying AI to medical software will ease these boundaries. By automating those internals, we’ll be able to lower the obstacles without compromising on privacy controls, organizational limits, and traceable responsibility we desire.

With those obstacles gone, individuals with richer historical datasets will benefit immediately. The value of a scan, test, or medical record isn’t limited to an immediate concern. It can unlock a future insight. A future diagnosis informed by history is both more accurate and timely.

This cumulative value is what the medical community misses when they downplay the ambition of inexpensive, pervasive testing. Looking at a single patient with a single concern, an MRI (with its comprehensive snapshot) might always seem preferable to an ultrasound. But imagine an ecosystem where most patients have a baseline full-body ultrasound history, a 50-cancer screening blood test, and ten other sets of low-cost, routine diagnostics. In this ecosystem, there is a high probability that these data points will unite to form insights that a purely symptom-responsive history never could.

Individually, each cheap test won’t be conclusive enough to move a diagnosis from “possible” to “probable.” This introduces a valid concern: a test that shifts your probability of a disease from a background rate of 0.5% to 2% doesn’t merit invasive testing, but it might trigger hypochondria. But the combined history of three different inexpensive tests can shift that probability progressively—from 0.5%, to 2%, to 5%, to 15%. At 15%, invasive testing is merited. Without that cumulative history, your doctor would have rationally recommended against it, potentially missing a crucial early intervention.

Inversely, cumulative data can help avoid unnecessary invasive testing. Suppose you present symptoms that trigger concern, but your extensive testing history lowers the probability that the cause is malignant. If the symptom alone suggested a 15% risk, but your historical data shifts the overall probability down to 5%, your doctor could rationally recommend treating the symptom or taking a path with less upfront risk.

Some may interpret this as a personal call to build their own medical histories at any cost. While not entirely unreasonable, that is not my point. I’m not just thinking about individuals with enough disposable income to afford boutique medical testing1.

Instead, I am defending the systemic value of testing that is cheap enough to be pervasive. Pervasive, low-cost testing can transform healthcare from a reactive, symptom-driven model to a proactive, data-driven one. It personalizes preventive care and provides the volume of data necessary for AI and modern analytics to spot macro-trends and micro-anomalies that a fragmented system would miss. Efforts to drive the cost of testing down to the point of pervasiveness are the foundation of that future. Those efforts should be loudly applauded, not feared.

The concerns about fueling hypochondria and over-worry are real, but ignoring the value of cheap, pervasive testing is not the solution. We already manage hypochondriac tendencies in medicine today. I wouldn’t call myself a hypochondriac, but I’m not immune to that type of feeling. In my experience, one anxiety-inducing event is the annual physical itself: cataloging every new bump or brief pain, anticipating the doctor’s questions, and overthinking the answers. We don’t suggest abandoning annual checkups just because they cause anxiety. Instead, we rely on a doctor’s bedside manner—a core component of medical training—to manage that stress.

I generally feel better once I’ve completed a checkup. I expect the outcome there depends heavily on how effectively the doctor communicates. That is where we can manage the risks associated with an influx of new medical data. We should expand our thinking beyond just the bedside manner of individual doctors, and focus on the design of the entire patient communication system.

The medical system already does this to some extent, but as we enter an era of pervasive data, there is a need for more. The answer to the anxieties of the information age isn’t to reject the data, but to design a healthcare system capable of communicating its true value.

AI is more than software, but software is very much at the core of AI. And the reality is the public has not been very interested in understanding software development as a profession. A point of evidence for this, and a mechanism leading to being poorly informed, is that software development is one of the least-represented occupations by entertainment television and film.

I made this point in a story a few weeks back, in a longer post about bubble narratives. It was part of a larger narrative, but I heard confusion about why I brought it up. I thought it’d be useful to pull this out, and clarify why I think it’s an important topic.

Since software is at the core of AI, it feels like people should be informed about it. Being informed requires more than being a user. It requires understanding the lives of those developing it. It’s one thing to be disinterested in how your phone or computer works, but happy to use it. It’s another to have only caricatures of software developers to rely upon, know little about the basic processes we use to secure, develop and deliver software, and yet have an opinion on AI development.

Here is what I wrote:

Why the public has a poor understanding of software development

The wider world has never shown broad interest in learning what software developers do. Compared to other professions like police, soldiers, doctors, lawyers, musicians, writers, journalists or even criminals. Without that interest it’s unlikely to learn the inner workings of the profession.

Media portrayals of software developers are rare and rarely accurate. The most common portrayal is the “hacker” who mysteriously takes control of computer systems in a few minutes with no preparation. Not only is that a poor representation of a real hacker, it tells you nothing about software development overall.

I reworked the data above to make it easier to see the patterns.

Ultimately, my concern here is deeper than portrayals. The facts about portrayals are an example. That example does have real effects, but entertainment media is also reacting to interests that precede entertainment choices.

The real concern is that advocating for laws or regulations that apply to software without an understanding of the process of development, is a recipe for failure. There are many places one might learn about an occupation. Media is just one, but it is a powerful one. Also, my experience as a member of the occupation is that most of my conversations about the occupation were limited to people inside.

There is a personal side to this. I’ve often been troubled by how the world in general treated the work I did as somewhat foreign. I noticed an imbalance in how much time conversations dwelled on my own work. It’s not the only profession like that, but it did register with me personally. I bring up my personal experience, not because my feelings are the point. They might be worth a little thought, but the point is I can see this disconnect so clearly because it is personal. If it’s not personal, it’s an easy gap to miss.

Software isn’t the only underrepresented occupation, but it is high stakes amongst those. It does appear in the news, but is it the facts of software developers that are being represented in the news? That’s not my observation either. The personal opinions of writers, artists and media that are heavily represented instead. Those same groups are over-represented in entertainment media, and have their opinions as the most visible via that route.

I’d suggest this is a moment where it makes sense to lean in a bit. It might not be natural to be interested in how software is developed. It might be more convenient day to day to enjoy the software, and treat the field as opaque. But if we need to make decisions about the field, that gap will become relevant. Everyone should be part of making those decisions, but the process of making those decisions must be well informed too.

So, if you’re a journalist, you should be interviewing software developers more often. If you’re creating a movie or TV show, you should consider writing in a character from software development, and avoid applying inaccurate cliches.

I’m trying to do my part with this writing. I probably can do more to make it more accessible. There is a tendency in software development, like any field, to write for your internal audience. That’s reinforced when outside interest is low.

If you’re a regular reader of this Substack, it’d be redundant to suggest you need to read something from a software developer, but I’ll assume this message may reach those beyond that scope. So encourage others to connect with the how of software development, both here and with other software developers in their lives.

That’s not going to be enough to bridge decades of disengagement, but it is a start. Since the political decisions of AI can’t wait for a repair, I suggest some humility in decision making here. In AI Safety Is Underfunded by Design, I suggest that we need balance that takes advantage of organic efforts of self-regulation. I wouldn’t say that self-regulation is sufficient; even if it’s working so far, this is too important to take that approach. But at the same time, regulating what you don’t understand will fail.

A learning process is necessary, both amongst the wider public, and with the politicians, lawyers, and administrators that would need to be part of any effective regulatory system, alongside industry experts.

Related Posts

The architecture I outlined had 4 layers: compute supply chain, operational infrastructure, intelligence and application. Each layer has to justify its own expenses, which contribute to the revenues of the lower layers. When those aren’t justified, the whole structure is a gamble on the outcomes of the upper layers. That said, the immediate term for lower layers can be remarkably sound, because they get paid now, not later.

This story is only half written, so we can look at how each layer is doing today. Even when that story looks good, we shouldn’t extrapolate to the future. Each increment of demand must prove itself, both in terms of feasibility and in terms of timing.

How is each layer doing today?

Layer 1: The Compute Supply Chain

The main influence on this layer is the inputs from Layer 2. Since those plans have kept progressing as planned a year ago, it should be unsurprising that this layer has done well. Revenues continue to increase and stability looks favorable. The main change here is that competition is growing. The availability of chips designed in house, TPUs, Trainium and more, have grown. The efforts to make Intel capable of competing with TSMC have continued. TSMC has continued expansions of its own.

Additionally, memory and CPUs have been pulled closer to the middle. Memory was already a bottleneck and has become more central. GPUs have remained central, but share much of the spotlight with memory now. CPUs have moved from trivialities, to moderate importance.

Layer 2: The Operational Infrastructure

The situation at this layer has firmed up significantly in the past 6 months. Revenue for delivered compute has continued to grow. Additionally, commitments have been disclosed, adding stability. Disclosures about commitment numbers don’t express the exact terms, so we don’t know if these could reverse.

More importantly, everything is broader here. OpenAI and Anthropic are using compute from all platforms. xAI showed how to recover from a failure, at least partially, by selling unused compute to Anthropic.

All that good news shouldn’t ignore that there’s still significant spending planned that will have to justify itself. The revenues of today (+$100 billion ARR) are roughly proving that last year’s $410 billion in spending isn’t going to be unproductive. That doesn’t tell us enough about this year’s $700 billion, nor next year’s $1 trillion. So long as those keep growing quickly, the risk that expected revenue to cover it never appears should remain a live discussion.

Layer 3: The Intelligence

There are many positive updates here. The best ones concern Anthropic, which is a moderately bad story for OpenAI. Revenues have grown considerably. Anthropic expects to be profitable. Enterprise revenues have grown considerably. Could this reverse? Maybe. Much usage is still prototypes, experimental, or otherwise dependent on expectations. But it seems unlikely. As I covered last week, there’s more pulling us forward than back here.

Layer 4: The Application

This layer still contains a large amount of uncertainty. The most significant use case, software development, is fundamentally another layer. Last week I covered dynamics of token usage by the application layer. While this is one of the longest articles I’ve written, it only covers dynamics, not actual numbers, and only a few of the most important.

To be truly solid, the various parts of the application layer need recognition that value is being created. That’s hard, because there’s a lot that would need to be measured here, and many of the things you want to measure, are both difficult and emerge with lagging indicators.

Adversarial usage, in marketing ($660 million), legal ($650 million), and sales ($390 million), is following behind coding and IT usage.

Early adoption from sales and advertising, shapes how much effects need to be proven. From a financiers perspective, this isn’t a problem, maybe even positive. From someone worried about employment demand, it also isn’t a problem.

But from a social perspective, it’s not sound. We can’t expect significant positive value to come out of adversarial sectors. We want to see demonstrated productive revenue, and we want to see cases where freed labor is reutilized in growth sectors.

Conclusion

Is the AI industry past the “bubble” discussion? No. Certainly not in terms of stock prices and valuations. In terms of CapEx spend, and the likeliness of future contractions or firm failures, it’s on firmer footing, but the story still progresses as the mountain is only half climbed. You might say that anyone who put 2026 as a specific timeline for their concerns, has missed the mark, and there are many who did. But overconfident predictors don’t make an effective counter-argument to more general views.

Related articles

Preview

It would be useful to know the shape of future AI demand, and many are attempting to predict that. Since this is a long piece I’ll give you my predictions up front. My overall prediction is that localized corrections, from the imposition of usage controls and consistent pricing, will ultimately be less important than the big trends. Value, so far hard to measure, will become more clear, first through incremental gains at the core of software development, and next, from the innovation that takes longer to accumulate and organize.

Read on to learn how I add my experience in cloud computing and software engineering to my deep interest in economics to extend responses from two of my favorite writers. Along the way, I’ll recast the bubble analogy, explain recent trends that have hit the news, explain trends hidden deep in the development lifecycle, and provide a model, “Ingenuity Matrix”, for mapping usage intent to expected outcomes.

We love good stories, especially those with a villain. But we should be careful about our stories, knowing how powerful they can be.

Three stories have hit a crescendo at about the same time. Tokenmaxxing — companies turning token usage into a goal, metering it, and the waste that incentivizes. Subsidized tokens — questions on the relationship today between AI costs and pricing. And under both, the doubt about whether spending is producing value for AI customers.

Stack them together and a tidy narrative falls out. If these are what’s driving token usage, and they all adjust at once, the readjustment will ripple through AI industry economics — including Anthropic’s recently skyrocketing revenues. That narrative extends across all model providers, culminating as a cascading failure of the whole AI industry. Call it the token bubble, brought on by a revaluation of tokens and their utility.

It’s a neat story, but the framing is off, even before we get to evidence. “Bubble” as metaphor smuggles in two assumptions: that we’re looking at one structure, full of only hot air, and that it ends by popping. In reality, industrial bubbles deflate, running out of air. There is an inflated shell, inside of which a structure is being built, and its collapse while deflating halts construction within, and damages unfinished construction. But something remains.

Inflating the shell is not folly, but the simplest path to enable construction. It’s still calamitous when it deflates, but the goal is the structure, not the air. So the question I’m interested in isn’t “are we in a bubble?” It’s: which of these dynamics is air, which is structure, and how would you tell them apart?

The story has been covered by two of my favorite writers, Derek Thompson, in The AI Boom Has Entered Its ‘Wait, Is This Worth It?’ Era and Noah Smith, in How much more software do we really need?. Both play speculatively with the idea that spending and rationality may have split from each other, but retain optimism that something worthwhile is being built.

Thompson concludes his summary of an interview with SemiAnalysis’s Doug O’Laughlin:

Every new technology requires an extended period of trial and error, as organizations toggle between (a) not enough experimentation or spending, followed by (b) too much experimentation and spending, followed by (c) too dramatic a pullback, followed by (d) the repetition of steps (a) through (c), until firms figure out a long-term balance between labor spending and tech spending. Whether AI skeptics like Marcus are right that the bubble is about to pop depends entirely on a question that, as of today, nobody can definitively answer: Is the bill worth it?

Smith considers the period before a smarter than human in all ways artificial general intelligence:

But until we reach that point, it’s a nontrivial task to think of business models that could be fully automated even with an AI that can’t yet do everything. That’s going to be hard! If I had any good ideas for how to do that, I’d go become a billionaire myself.

At some point, though — maybe in the very near future — people (assisted by AI) will come up with those revolutionary new business models. At that point, tokenmaxxing will suddenly become a lot more economical, and Anthropic — or whoever has good coding agents by that time — will stand to make untold amounts of money.

These are good perspectives, but I can improve upon them to help understand the dynamics of AI usage. First, I’m from the software industry, which is at the center of the maelstrom — coding is now the single largest category of token usage. I can describe in more detail what developers are actually doing with these tokens, and their motivations. These details are important. Without them, a lot of valuable work remains mysterious, which invites doubts, such as “is this worth it”, or “do we need more software”?

Second, I’ve spent a while thinking about the adversarial dynamics of some AI usage, since first writing about it last year. Those dynamics are key to the questions both writers leave us with. Adversarial usage doesn’t produce the social value we all seek. It is not the only driver of AI usage, but when it is a driver, we should be asking, “is this worth it”?

Both writers are aware of an important detail, timing, which explains many misleading observations. With the addition of a deeper understanding of software development, and that model for separating zero-sum jockeying from the creation of social value, we can recognize events along the timeline with more accuracy.

Token usage, like human labor, can’t tell you progress. Its best analogy is effort. If you want to understand the effectiveness of effort, you want to know how it’s being applied. Different applications correlate with different outcomes. Since you can’t fast-forward to the results, this is the best immediate categorization you can add. I call this categorization, the Ingenuity Matrix, describing the scope and social alignment of token usage.

Some token usage goes nowhere by design, some burns down a backlog of long-deferred work, some is zero-sum jockeying. A slower, quieter share is the significant work that actually changes lives. Sort the usage that way and the “is it a bubble” question dissolves into a more useful one — what’s being built, what events can we expect along the path, and what risks and opportunities come with each set of events?

Background

Before we start into the model, understanding the two terms behind the narratives is useful. This will also be useful when reading general news on the topics. The narratives on these conflate multiple meanings, and smuggle assumptions. That ambiguity can support misleading narratives1.

Why Bubble as an analogy is over extended

I said in the opening that using “bubble” as a metaphor for the AI industry smuggles in two assumptions. A bubble is so commonly used to analogize industrial revolutions, that we fail to reflect on the limits it has as an analogy. One mistake it leads us to, is the belief that there’s a soap bubble floating in air, and when we prick it with a pin it will pop, and evaporate. This does a poor job of explaining reality though.

We might limit our imagination more effectively by replacing the soap bubble with an inflatable dome. Whether this stays inflated depends upon the balance of air entering and exiting. Inside this dome, we’re constructing something durable, but it would be a challenge to do so with the dome weighing on top of us. We need the air to keep the dome’s ceiling from impeding our construction, and if it deflates it will probably ruin any half constructed structures. The stronger completed structures can sustain the weight of a deflated dome, but will struggle to conduct any additional construction.

If you want to think of the social support for a system, which supplies the air to keep the shell inflated, as a bubble, that’d be fair. This can evaporate with a bad news story, or some other form of social contagion. That social support is what replaces the air that leaks out. We’ll discuss the leaks later. Some are necessary, some are not. But replenishing the loss is unavoidable.

It’s useful to remember that in this analogy, deflation isn’t free. Something will remain, but the damage to unfinished construction is real. Careers are an obvious example of the consequences. When companies downsize the skills, connections and tacit knowledge built to support growth get stranded. If people move on, they may never come back. And besides, they are people and the disruption to their lives matters too. Projects also take a hit. Some projects may be zombies, shambling along with an unsound structure that will never be completed. But the forces of deflation aren’t so selective, and promising work is wiped away as well. Many projects that stop work during periods of tightening never start again.

A second flaw in the analogy is as a singular structure. Not only are there independent structures being built within, there’s not a single dome. There is a primary dome, where the model providers, GPU manufacturers and designers, and much else reside. But AI is also working to serve many different industries, and we shouldn’t assume a shared fate between all those efforts. We do want to pay attention to software development, because it represents such a large fraction of current usage. But software development itself isn’t an end of its own, it serves other industries. If AI is effective at helping some of those, and less-effective in others, this doesn’t establish a shared fate. It is only those cross-cutting effects that affect all software development that would carry that risk.

For the most part, those outside of software development aren’t going to understand those cross-cutting effects. I’ll highlight some of those details here, as they should be relevant to anyone interested in the immediate future implications of AI.

Why the public has a poor understanding of software development

The wider world has never shown broad interest in learning what software developers do. Compared to other professions like police, soldiers, doctors, lawyers, musicians, writers, journalists or even criminals. Without that interest it’s unlikely to learn the inner workings of the profession.

Media portrayals of software developers are rare and rarely accurate. The most common portrayal is the “hacker” who mysteriously takes control of computer systems in a few minutes with no preparation. Not only is that a poor representation of a real hacker, it tells you nothing about software development overall.

When software is “done”

If you come from outside the software world, you’d be excused from thinking of software development as building new software. In reality, this is a modest part of software development. Maintaining software, deploying software, and operating deployed software all represent larger segments than new software. All said, new software could be as small as 20%.

Noah makes a tentative argument that “The world may already have most of the traditional software that it needs.”. Noah’s aware he might be getting this wrong, and indeed he does. It does take an immense amount of work to keep sites running. AI is being used here, but it started later than its use to create new software. It’s not too hard to guess why. Creating new software is low risk comparatively. Like everyone, trust of AI has been a process. Software maintenance and operations themselves rely on significant “tech-stacks”, which have to be modified before you can even attempt to use AI to make a site more reliable in a meaningful way.

The number of software releases for security, operational, monitoring and development oriented features has been significant over the past year. Many use AI. Probably many others were built using AI.

Should you expect faster load times and higher reliability? First, would you really know? These have been improving for years, yet the general public rarely comments upon it. Mostly the only comments are those times when something does fail.

I’m not holding this data above as proof that AI has improved reliability. These improvements are more likely the result of conventional engineering, some started years before the results. The results that are “AI” based, are the result of the “machine learning” form that predated the architectures for Claude, Gemini and ChatGPT.

The point is, Noah (and you too) probably aren’t a sound judge of whether improvements are occurring unless you take the time to gather data. From my own knowledge, I know most AI based improvements are still in the early phases of adoption. But the average person shouldn’t expect to have an intuitive grasp on this. We’re bad intuitive judges of background effects like this, where we have to compare changes over time of non-continuous events. We can recall the last event, and the last change, but we are just as likely to draw a pattern from a recent reaction, than from an accurate history.

Why “tech-debt” comes first

You’ll find an interesting pattern that I’ll get into later. The first work to be done is the “shovel-ready” work. It’s easy to generate a prototype for some random idea, but rarer to have a great idea that can go from ideation to production quickly. AI does speed that up. But it doesn’t speed all work up.

With that in mind, provide a tool to a software developer, and they’ll have a long list of things they wanted to do, but haven’t had time for. Our general term for this is “tech-debt”, but realistically, it also includes half-baked feature ideas, or features that were sound but never made the cost-effectiveness cut. This list predictably contains a lot of non-amazing things. If they were amazing, they would have made the cost-effectiveness cut the first time. But AI does give you a reason to go deeper into that marginal backlog.

Security as a priority

I should also mention security here. Security is extremely important to the operation of software. Failures of security are nearly the worst thing you can imagine. This applies to all phases of software: development, deployment, and operations. It’s tempting to think of security as something you simply develop. But in reality that’s just the first step. A significant failure in development is likely to lead to a significant failure later, but it’s not destiny. You can layer protections to mitigate a development failure during operations. You have to do this because there are development failures you don’t know about. And more importantly, even a soundly designed and developed system can fail if not operated properly.

A lot of time and money is already spent on security. It’s never been the case that it hasn’t been a priority. You can find cases where it wasn’t a high enough priority. But it’d be a stretch to suggest there was a case no one cared. Whatever the priority, there is a limit, a cost-effectiveness barrier where one of the stages of development could have achieved more with more inputs. The introduction of AI changes the math on that barrier and makes many things practical that were impractical.

Security has another dimension too, which is that in addition to AI altering the developer’s cost-effectiveness equation, it does so for attackers too. This creates another incentive to burn down the security backlog. Security can’t wait. And so a lot with good cause, a lot of AI based productivity is going into security efforts.

This isn’t an effort that’s particularly visible to the outside world. What the outside world knows about it comes mostly from stories, not direct experience. When developers patch security holes, their intent almost always is to not change the user-experience. When that is the intent, it’s a slower process, because it requires educating users about new security mechanisms they need to participate in. Because that’s such a difficult thing to do, security teams have a very strong preference toward solving problems themselves without involving the users. It’s not always possible, but 90% of security efforts are invisible to users, and the next 9% are delivered as patches users see installed, but don’t pay any attention to.

The ingenuity matrix

I said in the opening that token usage is like effort: it tells you activity, not progress. To get from effort to expected outcome, you have to ask what the effort is for. Two questions do most of the work, and together they form a grid.

The first question is social alignment. Does the work create value the world didn’t have (positive-sum, pro-social)? Does it merely move value from one party to another (zero-sum, non-social)? Or does it destroy value — burn resources, or actively harm (negative-sum, anti-social)?

Alignment can be informed by our guesses of actors’ intent, but it’s not dependent on it. Our best bet is to act as an outside observer, guessing at outcomes. I don’t want to overcomplicate this though, this is estimation after all. Some significant pro-social value sometimes arrives from someone tinkering purely for fun. The social alignment is still recognizable from the outside, even when the actor wasn’t aiming at it.

The second question is scope, how far the work is reaching. Significant work aims at a real leap. Simple work aims at something bounded and modest. Naive work isn’t aimed at a productive outcome at all. Here “naive” describes the absence of a useful target, not the absence of a motive. Intentional waste is naive in this sense, it produces nothing of value, even though the person doing it has a very clear motive.

What you’ve just toured, security patches, reliability work, performance and cost tuning, is real value, almost all of it invisible to the people who benefit. Nearly all of it lands in a single cell: simple, positive-sum. It’s illustrative that so much of what is immediate is within simple or naive ingenuity. The first things individuals use AI for aren’t the significant ones. It’s the modest, shovel-ready, often-unseen things.

Map the rest against those two axes and you get an ingenuity matrix:

Negative-sum is not hypothetical, it connects back to security. The same drop in cost-of-effort that lets defenders finally burn down the security backlog also lowers the attacker’s cost. AI-assisted cybercrime is simple, negative-sum ingenuity, and the prospect of AI-scale biological or infrastructure attacks is the significant version. A large share of the invisible defensive work isn’t optional improvement, it’s the response to an adversary. The result is effort that is no longer avoidable, but also hidden, which delays the visible gains we’re watching for.

Significant non-social ingenuity ends empty. Non-social work can seem significant when under development. But one of two things happens. Either the work ends up leaking into pro-social, or anti-social accidentally, or it is copied and becomes trivial. Significance and neutrality are generally unstable.

Naive ingenuity is where the most visible tokens are burning right now and the least is being built. Failed experiments and aimless prototypes aren’t worthless — they build skills and occasionally surface something real, which is the option value the leaderboard bet was buying — but as a category they go nowhere by design. Because naive usage is so voluminous, and personal, it’s the most visible to the simplest forms of observation. That helps it dominate the “is this all a sham?” narrative.

Simple ingenuity has significant usage, but is quickly forgotten. The high volume usage is generally operationalized, contributing to security, reliability or operational efficiency. It’s soon forgotten, as it becomes a background effect. It doesn’t have the humorous, villainous story of tokenmaxxing. It doesn’t receive the personal promotion of the latest experiment.

One of the hallmarks of simple ingenuity, is it could be described as a backlog. The work may have been identified as desirable a long time ago, but with other competing priorities, it wasn’t prioritized. It may also not have been cost effective. One of the changes that AI brings is a change in cost-effectiveness. This activates this backlog, and you should expect early effects to burn this backlog down.

Simple ingenuity comes early and makes existing work more efficient. Sometimes this will show up as measurable revenues, but much is internal to companies. In that case it’s the token usage, the lower labor costs, or the higher quality that are the observations.

When AI enabled workers have a clear backlog, efficiency gains will flow into simple ingenuity to burn down the backlog. If the backlog results in priced or measured output, you’ll know.

Significant ingenuity will take longer to be identified, developed and deployed, especially the pro-social variety. The economy will reuse freed labor to create more value. That won’t happen immediately, as it may wait on hiring processes, training processes, or even the formation of new companies pursuing new products or business models.

Timing

While the development process is accelerated, the identification process retains most of its bottlenecks. Optimism may accelerate it. Idleness may accelerate it. But optimism and idleness may also flow into naive ingenuity, pursuing trivial goals without positive utility. There is a blurry area where naive ingenuity is experimentation. It may fail, but its failure may be necessary to build skills or discover significant opportunities.

At some point, a few things start to coincide. Naive and simple ingenuity will have built skills, ready to be exploited for realizing significant ingenuity. The backlog’s distraction fades as it burns down, and a new equilibrium raises the incentive to chase significant work — significance always carried more reward, but also more risk. But as cost-effectiveness decreases deeper into the backlog, avoiding risk becomes less attractive. All of these, in addition to the passage of time, predict a future wave of significant ingenuity that direct observation of measurements would fail to predict.

New output

Most of what we’d recognize as new output is significant, pro-social, and lagged. These are the life-changing things, and they’re the hardest to forecast. Your best guide might be a science-fiction novel, but of all the futures sci-fi writers have imagined, which do you bet on? Like flying cars, some things that look a step away stay out of reach far longer than expected.

It would be a mistake, though, to generalize from the failed predictions to all predictions. In many ways today’s information world already outruns older sci-fi imagination — the 1987 Star Trek: TNG depicted computers far beyond the 1966 version, and on the information front we’ve roughly met the standard it set for the 24th century already. The significant wave is hard to time and easy to underestimate at the same time.

What to expect

We should expect the AI industry to experience some pullbacks, then continue on. Whether this ever meets the bubble narrative is uncertain. I’m skeptical. Many pullbacks will be met by other accelerations. One experiment fails, another scales.

There isn’t one bubble, ready to pop, but multiple domes. Each industry, each set of users finds their value. While software remains so dominant, a failure in the software use case could be dramatic, but much of it is boring simple work that will continue to be automated for some time yet.

Much of the immediate term work is going to focus on the simplest, most invisible aspects. We shouldn’t discount the value there. Where it’s defensive, answering the negative-sum, like security, it has to be done. Where it’s part of more normal systems, it’s freeing resources, and developing skills and experience that will fuel more significant ingenuity in the future.

You do have to wait to see world changing effects. Software, as a model for implementing a workflow, will remain, and the general skills of software developers will be critical to this. Lines will blur, people will cross-over the lines, but ultimately the concept of software will continue to exist.

If the software dome does collapse, it will create structural damage, like all such events. Failed companies, layoffs, abandoned projects. Resources for naive experimentation would evaporate, and companies would proceed more cautiously. But a structure will remain. The burned-down backlogs that don’t un-burn, the skills that accumulated, the efficiency that keeps paying out, and the significant work just beginning to grow.

So, are we in a bubble? Will users and companies pull back on token usage, looking for value, discouraging wasteful tokenmaxxing? Will they react to pricing changes from model providers that close subscription loopholes that allow token usage in excess of what the same money would have bought per token via API? Yes, they will, but will that cause revenue drops that deflate the dome?

I don’t think so, there’s enough pending and developing work to fill the gap. Even if the significant ingenuity is still developing, the simple work is sufficiently valuable and important. But maybe those dynamics will return next year. If compute providers continue yet more expansions, they still might find them getting ahead of demand. There’s a lot of history to be written here. I’d just be careful about writing the ending first.

Sources

• Representation of professions in entertainment media: Insights into frequency and sentiment trends through computational text analysis; Baruah S, Somandepalli K, Narayanan S..

• State of AI, An Empirical 100 Trillion Token Study with OpenRouter; Malika Aubakirova, Alex Atallah, Chris Clark, Justin Summerville, Anjney Midha

• The AI Boom Has Entered Its 'Wait, Is This Worth It?' Era; Derek Thompson

• How much more software do we really need?; Noah Smith

• State of AI in the Enterprise The untapped edge; Deloitte

Related Articles

Avoidable Adversarialism

A job search is inherently adversarial within groups. Applicants compete against other applicants. Employers compete against other employers. But the relationship between an applicant and an employer is not inherently adversarial. There is a perfect match and those two matched pairs should want to discover each other.

When we talk about the adversarial nature of the modern job search, we should separate the inherent part from the avoidable part. The growing hostility between applicants and employers is avoidable. If we can avoid that we’ll collaborate. Job searches will resolve more quickly, saving money and pain. Workers will be happier and positioned to do the most good for themselves and their employers.

The devolution into adversarialism is clearly a consequence of limits to information exchange and processing. Employers started filtering resumes to minimize what they spent on hiring. Filters create something to defeat. If an applicant has a reasonable expectation that they’re a match for a role and they want it, they want to pass the filter. And since they may need to do this many times over, they have every incentive to do it efficiently.

The Historical Devolution of Hiring

Filtering predates computing. Long before anyone wrote a regex against a resume, recruitment teams read resumes by hand and made decisions from them. By necessity, those filters worked only with what appeared on the page, so candidates learned to structure a resume to pass them.

When resumes were reviewed by hand, there was some benefit to keyword matching. But there was a secondary method of passing the filter, weaving a story. A recruiter could lean on a strict rubric, or they could read for character, for an arc, for something that spoke to who the applicant was in a way that keyword matching couldn’t capture. That second channel encouraged investing in each application. Read the full job description, get to know the company, and write a cover letter specific to the role.

Human readers were also susceptible to word choice, which explains a lot of the fads in resume writing over the decades. Recruiters responded to certain modes of expression. “Synergistic” is the classic example, but the pattern runs deeper than any single buzzword. Applicants would notice an opening, exploit it. When recruiters realized they were being exploited a countertrend would set in.

The thing about that system, for all its gamesmanship, is that it evolved slowly. Trends and countertrends took hold one recruiter at a time. They might spread through HR conferences, trade publications, networks, but every update was individual. The arms race was real, but it moved at human speed.

Computerized Filters and Keyword Stuffing

Computerizing the filter degraded two things. First, the depth of processing collapsed. Matching became keyword-driven. Stories, arcs, and the patterns that needed a human to understand were no longer assessed.

Second, the motivation to invest in each application diminished because that investment was demoted to the second tier. If you passed the filter, then someone might read that and you’d benefit. But it was only rarely worth the effort with that extra distance.

This second part triggered an adversarial response. It became obvious to applicants that they were in a game, and that the game was everywhere. Callback rates fell from 10+% in the early 2010’s to 2% in the 2020’s. The decisions weren’t fair and weren’t optimal, so playing the game to its fullest felt like fair play. Resumes filled with keywords. Applications per candidate increased.

At first, human reviewers reacted by turning away obviously stuffed resumes forcing custom resumes per job with just the right keywords. Applicants had to balance to make these custom resumes not read like keyword-stuffed garbage to a person. This ate enormous amounts of applicant effort.

One page norms were initially strongly enforced. But then they gave out. The keyword filter wants more, because keywords need room to slide in while still appearing natural. Mainstream career advice shifted to two pages to leave room for the terms automated systems scan for.

The human reader, conventional wisdom holds, wants less, a clean document that doesn’t read as stuffed. But when a resume-writing firm ran a hiring simulation, recruiters preferred the two-page version by better than two to one.

Candidates were in a bind. Some filled resumes with 1-pt invisible white text, to please keyword filters and avoid reviewer reactions.

Then generative AI arrived and solved the bind. Applicants could generate a distinct resume for each role, loaded with exactly the right keywords, with acceptable writing quality, in seconds. But the story route was never revived, because there was no longer anyone reading for a story. Recruitment’s early stages became a machine with no room to absorb narrative.

Goodhart’s Law

There is a common description for that process, Goodhart’s Law. When a measure becomes a target, it stops being a good measure. Hiring teams have continued looking for targets to measure, but each has receded in usefulness.

Generative AI didn’t introduce Goodhart’s Law to hiring. It did remove the final exceptions. The cost of optimizing acted as a natural brake. Producing a tailored, plausible resume took effort, so people did it selectively. Now it costs nothing.

A single posting now draws hundreds or thousands of applications, because every applicant is applying to hundreds of postings. That drives employer response rates toward zero, which removes any reason for an applicant to invest care in a particular application, which pushes them to apply to even more roles, which raises volume again.

Detectors to the Rescue?

Instead of looking to solve the root problems, many hiring managers suggest they’d use another filter: automatically rejecting AI generated resumes. This is almost certain to turn away many qualified candidates. At best, it leaves behind those who invested the most time into the process. That’s a poor predictor of candidate quality. It entrenches gamesmanship too. If you’re tempted to make the argument that candidates investing more time have more confidence, consider the counter; your job may not matter to the most rational candidates. You may just be selecting for the candidates not smart enough to figure out the game.

Even the detector is suspect. Initially tools could hide the AI provenance, at the cost of quality degradation. Initially this was easy, then detectors got better. But even now, I can say, it’s not too hard to modify AI written text to not be recognized though. It would slow candidates down, but what’s the value in that to an individual employer?

While there’s some good advice to applicants to worry about accuracy with AI and resumes, this filter is low value. Turning away a quality candidate that used AI does not improve hiring quality. It won’t change candidate norms. At best it’s a guess that AI resumes are less accurate. That’s a questionable assumption due to the pressure on non-AI resumes to inflate.

More Requirements

So people propose fixes. But almost all of them share the flaw that they inflate candidate costs and arbitrarily filter qualified candidates out.

Consider degree requirements, which I wrote about separately in Or Equivalent Experience. Employers routinely demand credentials more restrictive than what their existing workforce holds, and automated screening makes that mistake worse, not better.

GitHub portfolios are a common suggestion. But building something good or great takes time. Most paid work ends up in confidential repositories. Those few jobs that produce public GitHub history benefit. For everyone else, it’s a free labor requirement.

You can spot an amazing GitHub a mile away, but the supply of engineers with that will be low. Categorizing good/average/bad is harder. Are you going to assess generally, or be tech specific? Most hiring focuses on tech specific skills. While I think first principles lead to more actual job success, they are harder to spot. If a candidate has a beautiful data-analysis project in Python, but an employer demands a message-processing stack in Rust, the candidate costs go up further. Any self-made project is about more than engineering skill, it’s also about ideas and time.

None of that is fatal, but the later gaming will be. If the signal is genuinely valuable and engineers are desperate enough, people will pay the tax. But what happens after it becomes standard?

Every standard encourages gaming. Simple things first, like using an underground agent to generate a unique, reasonable-looking GitHub project on demand. The escalation is next; hiring teams create a detector. Don’t expect the adversarial dynamics to stop at Level 1.

Any tool you hand to one side becomes a weapon in the arms race rather than a resolution to it. A new filter for employers invites better evasion from applicants. A better evasion tool for applicants invites better filtering from employers. One-sided solutions, however clever, pull toward competitive dynamics, because the other side never agreed to it and has every reason to defeat it.

My Experiences in Interviewing

Early in my time at Amazon (2015), I was active in the hiring process, conducting around 60 interviews in the first 2 years. I was surprised at how quickly I was pulled in. But AWS was hiring fast, and Chicago was a new office. The process had promising ideals. One favorite was the idea that your job was to draw positive proofs out of the candidate, not search out flaws. You wanted to hear them describe their approach to a problem or challenge that demonstrated their understanding of the right path forward.

I’ve found Steve Yegge’s writing compelling, and there is shared Amazon experience, so I was drawn to his recent The Last Technical Interview. Yegge was engaged deeper than I was. I avoided the bar raiser path. Imposter syndrome is one reason. I am proud of a number of people who I was part of their hiring loop, or mentored. If there’s anything that should give you good impressions of your ability to interview, it should be those. But I wasn’t feeling it at the time, and so when I found a technical problem to focus on, I latched on to that, and pulled away from hiring, doing a handful per year.

A less optimistic viewpoint was noticing interviewers that took pride in turning down +90% of candidates. If I approved 25% and they turned out well, what are the chances that those turning down two to three times as many weren’t turning down qualified candidates? There wasn’t any data to show their 10% was better than my 25%.

So I had to chuckle at the reflection where his team of interviewers voted not to hire 2/3rds of themselves. Made me feel a bit better about those nagging doubts that should be part of any difficult decision like this.

Interview Stages

The Last Technical Interview does get at something real though. His diagnosis: the interview has been broken for fifty years, even Google’s best interviewers couldn’t agree with each other or with their own past judgments. The whole apparatus is an elaborate attempt to generate signal, but consistently falls short. His prescription, the “campfire” model, is to bring people in to do paid, real work for a few days, then decide.

What makes his version more interesting than GitHub is a detail he calls counting the work twice: the candidate walks away with a permanent, portable record of what they did, stamped by the employer, whether or not they get an offer. If I get the concept, the employer would be doing a service for the candidate, and for all other potential employers. Both want the information a “stamp” would provide. Well done, this could counterbalance unnecessary adversarial tendencies that have accumulated.

Because it’s employer-certified rather than self-reported, it is harder to fake than a repo an agent can spin up. You’d have to find employers who hand out stamps to everyone, and Yegge is right that failure mode is self-correcting: a company whose stamps mean nothing has stamps worth nothing.

The network effect here is not one-sided. Both sides care about credibility. Gaming still exists, but if your stamp comes from a company vulnerable to gaming, its value diminishes. The signal lives or dies on credibility. Also, the more personal process offers fewer structural approaches to gaming.

There’s definitely some details left unspecified. I worry that some liability concerns could kill it. You still need to screen those invited to a “campfire”. You still have to scale.

An instinct of mine is that this requires some collaboration across the employer space. The campfire with credential stamp would do this, if it scaled. It would be challenging for every employer to create this credibility though if it’s by word of mouth.

An idea here is a company that does this as a service for an industry or multiple industries. This could solve the scaling challenge, if this is a passthrough. In some sense, this is what universities are: a 4-year campfire you pay tens of thousands of dollars to attend. I think the flaw is obvious, they are too expensive and too inflexible. In a sense though, they might have the best infrastructure for this impossible mission, should they choose to accept it.

The Shared Road Out

Which finally points at the answer to the question I opened with. If you want a solution that doesn’t just escalate the war, it has to be something both sides are actually happy with, or one side will fight it. One-sided efficiency tools breed counter-tools. A solution has to be a collaborative tunnel from the start, or it gets pulled back into the field of competitive options.

The trouble is we’re stuck in a bad equilibrium that’s individually rational. It’s a coordination trap, structurally a prisoner’s dilemma. A better equilibrium exists, where applicants apply selectively to roles they fit and employers actually read what comes in, and everyone would be better off there. But reaching it requires someone to move first and trust that the other side won’t simply exploit the opening. Right now nobody trusts that, for good reason. Applicants won’t invest in a tailored application when the expected response is silence. Employers won’t slow down to read carefully when they’re drowning. Both behaviors are sensible. Both perpetuate the trap.

So who moves first? Probably the employer, for an unsentimental reason: the employer is the scarcer and concentrated resource. A small show of goodwill from the side holding the scarce thing tends to get reciprocated. A well-known employer with a reputation can start something new. When candidates learn of it, they would opt-in. But it has to be something that builds collaboratively, rather than whittles away negatively.

I don’t have a clean answer. But I think the shape of the answer is clear enough, and it’s not a better resume or a better filter. It’s changing what each side gets from honest engagement, so that participating sincerely beats gaming.

Yegge’s instinct toward “gravity” is the right one, even if the mechanism is unfinished: make your rejection valuable, and candidates stop treating you as an adversary to defeat. A few partial paths point the same direction. Employer-certified, portable records of real work, if their credibility can be established. Skills-based hiring done deliberately rather than as language stripped from a posting. Two-sided interest signals like Greenhouse’s MyGreenhouse “dream job” feature, where a candidate can mark one application as special, though even that is still half a handshake until the employer offers something reciprocal. Something as basic as committing to send a real response.

None of these escapes the adversarial dynamic completely. Each one can be gamed at some level, and you should assume someone will try.

But that’s the wrong bar. The question isn’t whether a solution is immune to gaming. Nothing is. The question is whether it moves the incentives so that honesty is worth more than evasion to both sides at once. The inherent competition, applicant against applicant, employer against employer, isn’t going anywhere, and that’s fine. The unnecessary war, applicant against employer, is a failure to generate signal collaboratively.

Related articles

With that in mind, I have a reaction to seeing so many job postings in the software industry that look like this:

• Bachelors degree or equivalent experience

• 10+ years of experience building software

The first line is the requirement. The second line is a fig leaf — a way of technically not lying while still signaling what the process actually rewards. It was always a questionable practice, but AI is supercharging the impacts of this mistake.

The Logical Problem

Let’s start with the plain English issue, which is almost embarrassing once you see it.

From a logical point of view, these statements are redundant. 10+ years of experience will always be equivalent experience. There is no interpretation under which a decade of real-world engineering doesn’t constitute equivalent experience to a four-year degree.

The problem runs deeper than embarrassing logic. The reality is that equivalence is probably fiction. Most hiring managers, I’d wager, didn’t author this language and don’t think about it. But it activates biases in recruitment teams, offering a lazy shortcut, and sending the wrong message.

Automation and AI are Supercharging this Mistake

This was always a mistake, but it’s becoming more critical. Recruitment teams scanning resumes will be drawn toward an education section more readily than to calculating the equivalent experience. Automated tools in applicant tracking systems (ATS), including AI, have the same weakness, often more so.

How is a typical large language model (LLM) going to process these statements? The degree is a binary, well-defined data point with a clear answer. “Equivalent experience” is the opposite: fuzzy, context-dependent, requiring judgment. Even if the LLM evaluates the two statements correctly, it could make another mistake, treating the “or” as an “and”, or treating the two statements as components of an overall “closest match”.

These mistakes could cause a filter to fail, or it could cause a lower score. If the three clauses are processed independently, the degree holder gets 3 points, other candidates 2. If qualifications aren’t scored equally, one earlier in the list will usually get more weight. Even if a system doesn’t intentionally add a scoring system, a reasoning model could create one on its own.

More advanced systems are less likely to make these mistakes, but recruitment teams may not use the most advanced systems. This might be motivated by cost, or adoption started before systems advanced. They may even be non-AI, simple text analysis.

In this environment where job postings get hundreds of applicants, because every applicant is applying for hundreds of positions, a naive filter or scoring system can have a dramatic impact. The more nuanced aspects of a resume that should make you a top candidate, may never get processed. That means fewer interviews, dramatically reducing the probability of a successful interview.

What should you do?

The simplest thing to do is remove any such text from job descriptions. Since they are duplicative, and creating unintended effects, just delete them. And it’s not just those “or equivalents”. You should rethink degree requirements in general.

That’s not enough though.

If you’re not deliberate about this when working with your recruitment team, you may get no change or the wrong change. If they remove “or equivalent experience”, keep an internal filter or priority ranking, or just act on their own biases, you may get no change at all.

Should that be your intent? You might ask if you’re better off with the filter. There’s a couple things you can do to validate that this isn’t sensible. First, you might want to familiarize yourself with the actual rates of postings and workers. In many cases, the postings are more restrictive than the workers. Something seems wrong if it is true that if you had to rehire the entire workforce, 20% would be excluded. If 66% of those without a degree already doing the job wouldn’t get a job, you have to wonder.

Is that not enough? Then ask yourself, do you know something about the other 20% that makes you want to exclude them? I suspect you don’t. Once on a job, no one asks, but most assume.

Here’s the test: which of these has happened for you more often when working with professionals in their field?

• You learn someone doesn’t have a degree, and say, hmm, would have never guessed that! Bob is so smart.

• You learn someone doesn’t have a degree, and say, oh, now that explains it, I always wondered why Bob was so dumb.

It’s possible you have little data to work with, because people working on solving a hard problem don’t ask that kind of question. It does come up socially on occasion. If you do have a data gap, it’s not hard to close, just ask a few people. You’ll have to ask 50 people if they have a degree to find 10 that don’t. Because both the affirmative (no degree = smarter), and null hypothesis (no predictive power from degree), are on the same side, it doesn’t take a large same size to disprove the implied assumption.

The strongest case to put a degree on a qualification list is early in careers. This is where “or equivalent experience” actually makes sense. A candidate who just spent four years studying, 2 of it on practical development work is reasonably comparable to one who spent 4 years building products

But AI is hurting college grads. Should we give them an advantage?

There’s a pair of stories circulating about how bad recent grads have it in the job market. Should we give them an advantage, ensuring their expensive degree doesn’t come without rewards? There’s two data points cited on this topic, unemployment and underemployment. Both have flaws in their representation.

Underemployment

The underemployment data point is weakest, and generally just demonstrates a blind spot for those circulating it. The recent number is 41.5%, which does sound horrible without context. But all data should have context. This is not news, it’s a failure to understand the data. If I heard a number like that, I’d ask.. Well what is underemployment.. And what was it like in the past? It’s not hard to find the original source.

What you can see below is that 41.5% is lower than most historical periods. Is that worth freaking out over? No.

If you read the explanation, you understand why the number is this high. It asks people, working in the job, if a college degree is necessary. If you polled me about any software engineering, I’d answer no. I’m sure it is on the list, because probably the majority of the 80% of software engineers who do have a degree are answering yes.

While software engineering is likely on the list1 I can think of a lot of other things people commonly go to college for that even the college grads would likely answer no. Went to school to study art or music? Would you classify that as requiring a degree? Filmmaking? Social worker? What about some that might be on the list but are debatable? Journalist? Newswriter?

There’s also some majors on this list that appear because it’s tracking recent bachelor grads, but these majors usually go on to higher degrees (JD), law, business (MBA).

Unemployment

The unemployment story is more nuanced, but still represented as more than it is. It does show a change, but is it worth the reaction it has received?

What news stories highlight is that the light blue line (recent college graduates), has never been above (all workers) in the past. But is this a fair comparison? After all, a recent college graduate by definition starts unemployed. At some point in that 5 year span, they graduated, and started job seeking. It’s an apples to oranges comparison. All workers are the incumbents. Some of those workers may have been working the same job for the last 10 or 20 years. Since unemployment rates are based upon those “seeking” jobs, what you’re comparing here is first a fraction of all workers who lost a job recently, and then a sub-fraction of those who had difficulty finding a new job, vs. all college graduates, a fraction of which had difficulty finding their first job.

Shouldn’t “recent college graduates” be compared against “young workers”? There’s still a story there, in that the gap has shrunk, but the story isn’t that college graduates are getting a raw deal, but that there’s more equity between with/without. That’s a lot harder to make a decision about. How big a gap do we expect here? Don’t we want employment opportunities for those without a degree?

The Blind Spot

Partly I’m calling this out because it’s a current story that people are getting wrong, but partly I’m also demonstrating a general blindness that seems pervasive in what I’ll assume are mostly college grads discussing this story. They assume that the worlds of college grads and non-college grads are so universally distinct that there would be no overlap here. They assume that the only way you could be prepared for a job fit for a college grad is the same path.

The reality is that in terms of learning, college is just a convenient path, with a lot of resources laid out in front of you, no other responsibilities, and encouragement to follow a plan. College is many other things, a credentialing mechanism and an opportunity to build social networks, for example. But in terms of learning it’s not magical. You learn by consuming information and solving problems related to what you’re learning, and that opportunity has a lot of entry points.

Conclusion

College should be valuable to those that go. But its value should always stem from the learning it enables. Learning comes from many sources, the college experience is simply a well-resourced and well-structured source. Job experience is valuable in its unique way. In both cases, you have to make those experiences count. Your curiosity, your interest, and your hard work are what translate experiences to learning. A college experience, when done well, should be able to do this more effectively than a job. This is because enabling learning is its primary objective, whereas job experience has to compete with other objectives.

All that said, laziness will make any experience intellectually unrewarding. It’s worrying the degree of laziness applied to the hiring process and the data behind recent news stories. We should do better. We shouldn’t blame this on AI, that would be lazy too. Lazy slop was a problem before AI. It has a deeper cause. Maybe it’s growing, or maybe it’s always been with us. Whatever the case, honesty will get us farther than scapegoats.

Barriers, like degree requirements, enacted out of laziness or to create a condition of privilege are a mistake. We shouldn’t use them. In my opinion, degrees shouldn’t matter if you’ve already successfully done the job. This is doubly true if the job is more complex than anything schooling would have covered. In theory, it seems many employers agree, as the terminology, “or equivalent experience” has been common. But words are one thing, practice is another. A lazy translation of intent to practice that fails to meet the goal is harmful. This is but one of many, but hopefully I’ve made it clear how this one is a mistake.

Sources:

• Dismissed by Degrees: How degree inflation is undermining U.S. competitiveness and hurting America’s middle class; Joseph B. Fuller, Manjari Raman; Harvard Business School; 2017.

• Underemployment in the Early Careers of College Graduates following the Great Recession; Jaison R. Abel and Richard Deitz; National Bureau of Economic Research; 2018.

• Skills-Based Hiring: The Long Road from Pronouncements to Practice; Sigelman, M., Fuller, J., Martin, A.; Burning Glass Institute; (February 2024).

• The Labor Market for Recent College Graduates; Federal Reserve Bank of New York; 2026

Related articles

In general, market actors do not have great incentives to protect against catastrophic risks. They are massive negative externalities, often dwarfing the balance sheet of any individual firm. Say Anthropic releases a model that a malicious actor uses to conduct a cyberattack that does $5 trillion dollars in damage. Anthropic is only worth $800 billion, so if they get sued for $5 trillion, they are already well past the point of insolvency. A catastrophic harm may well already be “lights out” for Anthropic, or any other company, so there is little incentive to avoid them, if doing so entails real costs in the present day.

He’s right about the structure of the problem — but “little incentive” understates the precision available here. AI companies do have incentive to avoid catastrophic outcomes, just systematically less than society needs them to. That gap can be quantified, and quantifying it points toward what a corrective policy should actually look like.

The concerns he’s talking about — catastrophic risks — share a structural feature that distinguishes them from others: they are lumpy. A single catastrophic event, rather than a diffuse trend. The incentive is quite large, but not as large as it should be. These dynamics are worth exploring, as those ultimately shape if and how we structure a response.

Consider a hypothetical AI company, worth $800 billion. Now consider a hypothetical event causing $5 trillion in damages. If this event happened, that AI company would be out of business, so they have an incentive to prevent it. But how much incentive? The most they can lose is the whole company, so $800 billion. Since a lot of that is goodwill, in reality, losses become irrelevant earlier. For the sake of example, we’ll say $400 billion. If you had to pay half your market cap, you’re not worth $400 billion, you’re bankrupt, and worth $0. All claims greater than $400 billion have equal impact, since each produces the same outcome, a total loss.

This creates an imbalance between societal goals and the AI company’s goals. That imbalance could lead to underinvestment in safety, or risk taking that is out of alignment with societal goals.

We can quantify this imbalance, by modeling a damage cap in expected value calculations. If our $5 trillion event has a 1 in 10,000 chance of occurring, the uncapped expected value of avoidance is $500 million. With a damage cap of $400 billion, it’s only $40 million. Society should want that other $460 million in incentive to be shared by the AI company, but without an arrangement, it’s not.

Refinements

I used a simple model above, with linear effectiveness of investment in safety. It isn’t linear. In a linear model, spending $500 million reduces risk to zero, and $40 million reduces it to 1/12th of that, or one 1 in 9,166. But we could imagine, in fact we should expect, that the first $40 million does more than the next $40 million. Maybe the first reduces the risk to 1 in 100,000, and the next to 1 in million. It’s the same proportional improvement — 10x. But in the first case it reduces the risk from 100/million to 10/million for a total reduction of 90/million. The second case reduces from 10/million to 1/million, for a total of 9/million reduction.

To illustrate, I constructed a model that used logarithmic decay from the initial 1 in 10,000. In this model, under their default incentives, the AI company would want to spend $13.3 million to reduce their expected risk from $40 million to $8.7 million. But the societal risk is still $122 million at this point.

The goal of a corrective policy would be for the AI company to act upon the societal risk, which justifies spending $35.2 million to reduce the societal risk to $8.7 million.

Further refinement considers whether organic spending is more efficient than regulatory-induced spending. For example, if regulatory-induced spending had half the effect per dollar as organic spending, not only would the spending go up, but the residual damage would be higher.

In Scenario 1, all spending is equally valuable. In Scenario 2, the company spends efficiently up to its capped motivation, after which each real dollar buys only $0.50 of effective safety. And finally in Scenario 3, all spending is at 50% effectiveness.

You can explore other scenarios in the linked single page app (GitHub Repo). You can experiment with different decay functions, damage sizes and company sizes.

Organic Forces

We’d also be doing ourselves a disservice if we didn’t recognize the outstanding work that AI companies have done — Anthropic most of all. Commercial incentives alone aren’t sufficient. The stories about the founding of DeepMind and OpenAI make clear that good intent has played a positive role. But in a commercial world, you can’t depend on good intent to reliably show up or win internal contests. The pressure to go off track is substantial. For something this important, that’s a lot of trust. We want to use these forces, because they are efficient, but must not be naive either.

We don’t want to take organic forces for granted. If we assume they don’t need support, they might disappear. If we don’t acknowledge their value, we might strangle them.

Small Firms

The alignment problem becomes more acute for smaller companies. What if a smaller startup, with none of the weight of a larger company — little to lose, and everything to gain — rushes ahead, and skips best practices that avoid harm?

The leading labs are large (Anthropic, OpenAI and Google), but we shouldn’t take that for granted. Frontier model training costs keep going up, but the costs for a particular level of capability keep going down. DeepSeek proved that moats are much shallower than assumed.

You do want to avoid locking out startups, but also need a baseline that ensures safety isn’t skipped. A first step here is ensuring safety practices are shared. That lowers their costs in pursuing safety.

The current voluntary norm — leading labs sharing safety methodology despite having competitive reasons not to — is a favorable state of affairs that formal structure can preserve and extend. It will take organization to make it work at a deeper level. Sharing details of some safety practices publicly can add risk, so a well-trusted network for sharing enables more than just the public domain approach. Formalizing sharing as a condition of operating at the frontier, both preserves what already occurs, and can extend it more deeply.

Regulatory Shape

The model makes the policy objective concrete: close the gap between what the company is motivated to spend and the societal expected value, without crowding out the organic safety investment that’s already happening.

A naive response assumes insurance is enough, and the challenge is finding a large enough reinsurer to pay out. An even more naive response assumes this challenge can be fixed by inserting the federal government as a backstop to the insurance. The flaw in this thinking is that it makes society responsible for paying itself back for harm done to it. This won’t work. The harm would have been done. Society would pay for the majority of the consequences of the gamble the AI companies made.

These dynamics suggest that effective regulation needs balance, in order to use organic forces, and yet also not leave a gap. Dean is right that it improves the case for government involvement. If there’s a gap between the company’s incentives and the societal incentives on a topic so important, we should align those.

An industry body that both shares security practices and sets standards is a start. Shared excess liability amongst all AI companies would add to existing incentives. If one fails to prevent harm in a small way, that company fails alone. If one fails in a big way, they all fail. Expanding the pool in this way is better than involving the government, as these are the players with the ability to influence the risk. Those incentives will encourage maintaining quality standards, but keep the standards moored to efficiency and effectiveness.

That’s still not enough though, so a government body above that respects the value of organic forces, would be a second step. The challenge here is how to prevent this body from losing interest in efficiency. It’s natural for them to be interested in effectiveness, but efficiency comes with more difficulty. If standards ignore efficiency, you undermine the organic forces and risk taking a step backwards instead of forwards.

What doesn’t work?

The framework I’m discussing, can appear to be a compromise between two points of view. That’s not the intent. There is no intent to choose a middle point, in order to satisfy two points of view. I think the merits of this model fit without any politics.

The model does however balance multiple forces, and is not aligned with any maximal plan. That type of balance only makes sense if the maximal plans aren’t reasonable. To make it clear, I don’t support any maximal plans. It will take additional posts to flesh out why, and others have defended these points independently. But in the light of outlining my thinking, the basic is:

• AI bans: You have no chance. You have no global solution. It’s not a good idea in the first place, as AI will be very useful, but that’s not the biggest flaw. The biggest flaw is all of the partial wins - company X refuses to use AI, country Y bans AI - they all fail in the end and don’t contribute to any goal aligned with the best case for a ban.

• No regulations: Clearly something is needed here. This group is somewhat of a strawman though, as even people like Dean Ball see a role for regulation. The better critique is that there are many people who are implicitly “no regulationists”, because they oppose everything proposed and don’t put together enough to actually do something.

• Top-down regulations: Strangling organic safety efforts in top-down paperwork is a surefire way to fail. That doesn’t mean there isn’t a top, but it does mean, it can’t be total, and since it’s starting later, it should expect to start small and iteratively find its fit.

Clearly, there are more details to cover here. I’ve only touched on one dynamic that sets an overall tone, but you’d eventually need a list of initial best practices, and an expert-led group to maintain them. You’ll need a mechanism to choose that group, and a list of powers and limitations that define how they work together, and resolve conflicts. I’ll leave those questions for a future post though.

An updated revision of this article was published September 29th, 2026: Trusted Deployers: How to keep dangerous AI out of reach while preserving model choice

Preventing the use of AI for malicious purposes is critical. Malicious use means some human somewhere wants to create harm. AI is a new tool for them. In theory, existing law would apply to those creating harm.

Today I wanted to talk about some challenges that complicate preventing malicious use.

Jurisdictions

A first failure of existing law is jurisdictions. The world has rogue states, lawless states, and aggressor states. These either turn a blind-eye toward harmful activity, lack the capability to enforce laws, or actively create targeted harm themselves. Existing laws cannot reliably reach actors that hide in these jurisdictions. There is a justified effort to close those gaps. There is slow progress. Sometimes gaps reopen. Because it’s a long running effort, we shouldn’t expect a near-term resolution, and treat it as a reality we must mitigate.

If we can’t target the originator of malicious acts, we can try to deny them tools. We should recognize the efforts of AI companies here, which have been substantial. But, these efforts are hindered by two background stories: open models and privacy. To deny tools for malicious use, you must first detect malicious use, or intent; open models and privacy complicate both of these.

Open Models

Open models are models released openly. Without going into too much detail, the key quality is users can run these anywhere. Closed models don’t give users that ability, and users have to interact with them as a managed service. That layer of management provides the key capabilities that enable monitoring and denial.

Open models once openly published, have no or limited ability to monitor. There is very limited ability left to control them, mostly centered around denying access to sufficient compute resources.

The largest collections of compute are at cloud providers, but there are still ample compute resources outside of cloud providers — in private data centers, colocation facilities, sovereign national infrastructure, and increasingly, distributed consumer hardware. Even for cloud resources, the nature of providing compute, rather than a managed service obscure the most effective means of monitoring. By design, cloud providers give customers using compute a heavy dose of privacy.

While open models have their justifications, from the realm of preventing malicious use, they are a challenge. It’s of some comfort then that open models are less capable than closed ones. This reduces the capability harmful users have access to. Since some aspects are adversarial, the advantage of closed models provides defenders an advantage too. This applies most significantly to cybersecurity.

Will open models stay less capable than closed ones? We could, across cooperative jurisdictions, enact regulation to ensure that — but if a non-cooperative jurisdiction has the capability to create more powerful models, we’d lose that control. China is the jurisdiction most likely to both have that capability, and make independent decisions.

Privacy

The second background story is privacy. The default state of anonymity on the Internet has costs. Privacy advocates attempt to maintain this state. I, like some others, believe the costs of this anonymity as a policy are too high. This isn’t specific to AI, but it does relate.

We have tied the hands of security teams and mostly delivered theoretical privacy. Where privacy matters most, such as totalitarian countries, the privacy is undermined by local realities. Privacy advocates don’t have a voice here. They win political contests where there is the least need for them, and lose where there is the most. It’s a tough choice, but I think we’re not making the right choices.

We should be pragmatic, but we’re idealistic. In some cases, privacy measures accelerated accumulation of data for malicious purposes. When countermeasures can’t be due to obscuring the lowest layers of a technical stack, we fail to achieve privacy and prevent harm. When formal data-sharing is prohibited, informal systems take their place, and predictably result in harmful breaches.

If service providers always knew who was using their service, they’d be able to deny access to anyone detected acting maliciously in the past. But the internet offers too much anonymity. Providers can shut down an account, but without accounts tied to a real identity, a new one can be created. The current standard among AI companies is too lax about this. We could make it more costly for attackers to maintain access.

Conclusion

Jurisdictions, open models, and privacy are features of the world we must work within — but they are also policy choices we can influence. The uncomfortable reality is that these three forces compound each other. Open models place powerful tools in jurisdictions beyond legal reach, while anonymity makes it difficult to detect or deny access to bad actors even where laws do apply. Treating any one of these in isolation understates the problem.

The path forward requires accepting some hard tradeoffs. Meaningful identity verification will feel like a concession on privacy — because it is one. Regulatory constraints on open model releases will frustrate researchers and developers who have legitimate reasons to want them — because the benefits of openness are real. Coordinating across jurisdictions will be slow and incomplete. None of these are reasons to avoid acting, but they are reasons to be honest about what any given measure can and cannot achieve.

What’s not acceptable is the current default: deferring hard choices while treating anonymity as an unqualified good and open access as costless. The tools for harm are improving. The window for shaping how they’re governed is open, but it won’t stay that way.

Now, Trump and Republicans also try to just directly lie, suggesting gas prices aren’t high, won’t be high for long, and hey, this was all necessary. That type of lie only works directly with the most deceivable. It may help keep some of them from waking up, but the real purpose of that type of lie is to make the more subtle lie less obvious.

The more subtle lie, equating price spikes under Biden and Trump, doesn’t try to convince the public not to blame Trump and the Republicans for this spike, it tries to convince them to ignore it, suggesting, you still need to vote for us, price spikes would be worse with Democrats.

The reason no one should accept that argument is Biden didn’t cause those spikes. The simplest way to understand how poor the argument is, ask how much oil did the Biden presidency remove from the market? I can tell you, it wasn’t 15% of global supply. Probably not even 1%. I think you’d remember that story if it happened. It didn’t. Instead a different story happened.

When the pandemic occurred, global oil consumption dropped, and prices with it. Producers stopped prioritizing new supply. Later consumers started to return to prior patterns, and producers lagged behind the re-emergence of that demand. The price spike was a simple reflection of that, not a result of any US government policy that removed production.

The two major reductions in production since 2010 occurred under Trump. I wouldn’t blame Trump for the first production drop in 2020. It’s stupid to try and blame presidents for everything. But it also clearly isn’t Biden’s fault.The spike in prices during the rapid period of demand recovery shouldn’t be either.

Having some awareness of who made what choices and why is a better method than just direct association of prices to current governments. Instead it’s obvious the production drop was producers responding to the consumption drop.

Coming back to today, another ounce of blame that is deserved, is the destruction of the US electric vehicle industry, and more generally the entire clean energy industry by Republicans. EV Sales increased by five-fold under Biden, and have stalled/declined under Trump. If you want low gas prices, it helps a lot if your neighbors aren’t using much. Eliminating the $7,500 point of sale credit for these vehicles was a strategic and tactical mistake.

The on-off engagement with renewables is another strategic mistake that’s related. While renewables direct impact on gasoline prices are small, the impacts on batteries flows over into diesel generators and the EV industry. The big strategic failure of on-off engagement has been to fail to develop a valuable industry, allowing China to dominate.

Ultimately, the primary determinant of today’s prices is obvious. Trump’s war and its repercussions.

Last year, I started a series of posts on trust, with Money: More Than Just Stuff, It’s Trust as the focal point. I felt 1-year was a good time to update this account, and extend it.

If you ask an economist what money is, they will likely give you a functional, three-part definition: it is a unit of account, a store of value, and a medium of exchange. If you ask a dictionary, it will tell you that money is “something generally accepted as a medium of exchange, a measure of value, or a means of payment”.

These definitions are perfectly workable for daily life, but they contain a loophole. Defining money as “something generally accepted” describes a symptom, not a cause. It relies on the word “something,” anchoring our minds to physical objects—gold, silver, paper, or digital ledgers. But the link between an object and its status as money would be severed by a loss of acceptance.

The basic definition is useful, but it’s clear economists do not consider it sufficient. Just this week Stephen Dubner of Freakonomics, released Part 1 of a two-part series, What is Money?, covering an adaptation to an oratorio of Adam Smith’s The Wealth of Nations. Money must be something of a higher order than temporary objects we attach it to.

To understand what money is, we must look past coins and paper. Trust is not merely a social feeling; it is a vital component of cooperation that requires immense social energy to build and maintain. While trust can theoretically be extended freely, reliably creating it at scale carries a real, limiting cost. Money is humanity’s greatest collaborative technology—an accidental invention that survived and spread precisely because it lowered the cost of trust.

The High Cost of Trust

To understand why money is a technology of trust, we have to look at the world before it existed. How do human beings coordinate the exchange of goods and labor without it? Historically, humanity relied on two deeply flawed workarounds: Barter and Kinship.

The Barter Evasion

It’s common to think of barter as a primitive ancestor of money, but it is better to think of it as an attempt to trade without trust.

Imagine you have a surplus of hay, and you need milk. You find a farmer with milk, but he doesn’t need hay; he needs firewood. To make a successful trade, you are forced into a complex puzzle. You must find the person who has firewood and needs hay, trade for the wood, and then return to the dairy farmer. Economists call this the “double coincidence of wants.”

Because there is no trust carrying value across time—no mechanism that says “I gave you milk today, I owe you value tomorrow”—every transaction must be settled immediately, item-for-item. In a physical sense, barter avoids the need for trust, but the friction of searching for perfect matches makes it impossible to scale.

There is also the problem that the neat and tidy view of trust in barter being solved by the direct physical transfer of goods, is a bit of a myth. Anthropological studies of societies without money show trust issues relating to trust on fairness of exchange, both during and after negotiation.

The Kinship Tax

Because barter is so inefficient, early societies rarely relied on it for daily survival. Instead, they relied on kinship networks. Barter was used mostly outside kinship networks.

Early societies solved the trust deficit through deep, interpersonal relationships. You do not barter with your brother, your cousin, or your tribemate. You give them your surplus milk today, trusting implicitly that they will provide you with firewood next winter.

This creates a high-trust environment, but it comes with a fatal flaw: it is unscalable and exclusive. Maintaining deep, bilateral trust requires immense social energy. You can only maintain it with a small, localized group of people—a single-layer network naturally capping around Dunbar’s number of roughly 150 individuals. While societies can attempt to force kinship to scale by creating additional layers of hierarchy, each new layer adds complexity, instability, inefficiency, and immense human costs. This dynamic imposes what writer David Oks has called a “Kinship Tax.” It crowds out the ability to trust strangers, trapping economic coordination and human development at a deeply local, tribal scale.

The Technological Leap

Before money, humanity was trapped between two dead ends. We could choose barter, which offered zero trust and infinite friction. Or we could choose kinship, which offered high trust but severely limited scale.

Bilateral trust—knowing and trusting the specific person you are trading with—simply became too expensive to produce as societies grew.

Money was the technological breakthrough that bridged this gap. It allowed humans to substitute the expensive, unscalable trust of kinship for a cheap, highly scalable institutional trust. When you accept a dollar bill, a gold coin, or a digital transfer from a stranger, you do not need to trust the stranger. You only need to trust the token.

But how did this leap actually happen? It wasn’t designed by a visionary or decreed by a king’s master plan. Instead, money was an accidental invention. It emerged from the bottom up, sustained by stable local equilibriums where substituting a token became easier than finding a perfect barter match. Once stumbled upon, this system spread through evolutionary fitness at a societal level. Societies that adopted this scalable trust out-cooperated, out-traded, and out-grew those that remained trapped by the limits of the Kinship Tax.

Follow-up

This post has focused on why money is as useful as it is to humans, and why that usefulness is best described as an extension of trust. This falls into the Technologies of Trust series. While this first post updates on the value and concept, others will deal with history and deeper meaning.

This choice carries commercial, security, and community implications. However, as technology evolves, traditional arguments for both models require a second look.

Security Implications

Advanced security scanning, like Mythos, will create new motivations for keeping software closed source.

Historically, the “million eyeballs” effect ensured that open-source security defects were quickly found. But in an era of AI-driven scanning, the equivalent of a billion eyeballs can simply be purchased as compute tokens.

Consequently, the primary security benefit of open source declines, while the advantage of closed source—forcing attackers to probe a compiled “black box” rather than reading a blueprint—remains. We should expect the security balance to shift accordingly.

Commercial Implications

Conversely, AI tools capable of reverse-engineering software from specifications weaken the commercial moat of closed source. If a replica can be generated from behavior alone, the protection of hidden code diminishes.

We should be careful not to overstate those capabilities. While advanced AI tools can create working replicas in many cases, a simple approach to this will produce a less capable, less secure and less maintainable replica. And an advanced approach will require a lot of tokens (which you must pay for), and the efforts of someone who knows what software needs to be good software.

Still, even with those qualifiers, a shift occurs, and developers are left with a little less commercial motivation toward closed source. An open source software package that binds a community to it could be a more stable commercial decision.

The Background Shift

While both of these implications are interesting, they are both watered down by the shift toward managed software that’s progressed over the last decade. Software as a Service, and its variants (Platform as a Service, Infrastructure as a Service) involve a third-party taking responsibility for some part of the managing running software. Management provides a way to offer value beyond the observable parts of the software. In the realm of security, the privilege of management can be used to layer protections. In the realm of commercial implications, value may derive from the efficiency and organizational capabilities to operate the software well.

Those security protections allow providers to rely heavily on open-source repositories for foundational logic, but wrap those deployed software in managed, closed-source service layers. This intermediary role is crucial. It creates a secure boundary where security teams can insert active, AI-driven monitoring and take an adversarial role against attackers with the advantage of obscurity. By funneling interactions through this managed layer, threats can be caught and mitigated before they ever touch the raw, open-source code underneath.

In a sense, open-source both won and lost, as the dominant shift was not from closed-source executables to open-source repositories, but from close-source executables, to managed service deployments based on open-source repositories. The managed service layer provides many of the security benefits of closed-source executables, by allowing security teams to take an active adversarial role, with an obscurity advantage. By having some private tools and techniques, they could often have proactive responses to attacks, rather than only reactive ones. This overall mix, millions of eyeballs on the source, with additional managed layers has been a potent one, and will remain so. That said, we should expect some change in the balance here, with a greater part of the managed layers as closed source.

Conclusion

In the realm of managed services, I’d expect the net result to encourage doubling down on the trend. Proprietary layers to create well managed services will proliferate. Competition with open-source software will not be a priority, but proprietary forks and extensions that improve performance, manageability or security will be.

One question is, who will donate the tokens for scanning open-source repositories? You can’t expect open source developers to buy and donate tokens for scanning the same way they donated their time. Industry cooperation, sponsorship and coordination will be needed here.