AI, Determinism and Control (Part 2)

The Agent Ecosystem and the Human Hand-off

In Part 1 of this series, we explored how AI is fundamentally altering software control through the lenses of determinism and scope. We traced the journey from passive, strictly bounded chatbots to the threshold of active agents—AI systems capable of autonomous, multi-step planning. But what happens when these indeterminate systems are given broader scope and powerful tools? The consequences ripple outward, reshaping not just the security of our infrastructure, but the shape of our workflows and emotional relationship to work. To understand the recursive systems of tomorrow, we must dive into the agent ecosystem itself.

The Agent Ecosystem

Agents represent a significant shift in control, trading linear human prompting for continuous indeterministic planning.

To understand how these agents operate, we must briefly consider tools. Agents use tools to accomplish their plans. Tools can be anything, and which tools an agent is provided with define its constraints. You can provide an agent instructions, cautions, and directives through its prompt and context, but like anything in an agent, it’s indeterminate.

A tool might be as basic and low-risk as retrieving a specific account balance, where the boundaries are tight and predictable. It might be as broad as searching gigantic repositories like the entire internet or an organization’s internal files, which escalates risk by exposing the agent to untrusted data or sensitive information.

A broader path still is the ability to create and execute computer code, introducing severe risk if left unchecked. That might initially seem like it loses all constraints, allowing the agent to perform unanticipated or dangerous actions. However, code can be executed in a sandbox that limits how it communicates and what data it can access. Assuming the sandbox is secure—which requires careful planning, inspection, and testing—restricting communication to untrusted sites prevents data exfiltration or external control. Just as critical is controlling the credentials provided to the sandbox. Strictly limiting credentials restricts the agent’s ability to update records or access systems outside the purview of its current authorized activity. Together, these boundaries provide the necessary mechanism to constrain this high-risk capability.

Tool use isn’t restricted to retrieving information, either; it can allow changing information, which can trigger further actions. This is an area that requires much more caution, doubly so for writes and actions that are irreversible. Beyond that obvious observation, two other dimensions enter in. First, since an agent’s plan is indeterminate, the ability for a designer to remove the risk that it performs actions in unanticipated ways is vastly more complex than when working with a deterministic plan. Second, we must account for prompt-injection—the risk that something an agent has read can influence its choices, resulting in actions desired by an attacker rather than the user or designer. There are protections against this type of attack, but it would be foolish to consider them foolproof.

With that foundational understanding of how agents act on the world, we can observe this frontier opening up across escalating levels of scope:

Standalone AI Agents

Unlike a chatbot that waits for a prompt, a standalone agent is given a high-level objective, allowed to indeterministically generate its own step-by-step plan, and execute it using available tools (like searching the web or scraping data). While the planning is continuous and autonomous, the agent still typically operates within a relatively bounded scope, restricted by specific APIs to prevent runaway consequences.

Like chatbots, there are standalone agents from OpenAI, Claude, Google, and others. In fact, most chatbots have silently become agents¹, though still with many constraints.

In time, these standalone agents may have more and more autonomy. But from the perspective of this framework, the fundamental aspect of taking user input, deriving a plan through an indeterministic process, and executing that plan won’t re-enter the realm of determinism until it invokes a tool².

Agents Embedded in Applications

Moving beyond the simple “embedded AI node” discussed in Part 1 involves agents operating continuously alongside users within a shared software environment. Consider a complex data analysis platform: the human user might explicitly invoke deterministic tools to filter data, while an embedded agent operates in the background, autonomously invoking its own set of analytical tools to highlight anomalies. The application becomes a hybrid ecosystem where human indeterminism and agent indeterminism collaborate in real-time bounded by the application’s guardrails.

Agents embedded in applications have an advantage over agents called by other agents: the input data is controlled by the calling application. Still, remember that the applications agents are embedded in may themselves be working with dynamic data. A data analysis platform has many data sources; are they all vetted and invulnerable to an injection attack?

Another common example today is agents embedded into development workflows. They can reason about code, look for security issues or defects, generate fixes, and submit them as pull-requests for developers to review, effectively combining the code-generation function with the embedding function.

Agents Using Agents

An agent can become a tool used by other agents. To think about why this is valuable, you want to first understand that agents generally have a few components. At their core, they create plans via the GenAI model. They generally have some sort of instruction (or persona) file. They’ll also have access rights or boundaries associated with a tool list.

The instruction file is the interesting part here, as it provides the reason for a distinct agent. In the simplest version, it might describe a persona (“you are an insurance adjuster”), but this relies on the GenAI model to blindly decide how an adjuster behaves. If you’re building an agent, you want more control. A detailed instruction file can’t shift you to a fully deterministic world (if you want that, you should write code), but it can reduce variability.

Another aspect that can remove ambiguity is placing restrictions on inputs and input sources. If an agent expects to receive user input, it has to be fully prepared for anything. If it’s called from an application, those expectations are more constrained.

Agent input expectations fall into two categories: expectations formed around successfully fulfilling its goal under valid usage, and expectations formed around avoiding taking action on behalf of an attack. These two have some non-overlapping aspects. If an input is suspected of being for the purpose of an attack, there’s no need to try to do anything other than quit and refuse action. But the consequences of allowing an attack are generally far more severe. On the other hand, failing to successfully complete an action is less severe, but it’s less acceptable to give up because of uncertainty.

If we had wanted certainty, and accepted inaction for uncertainty, we should have written a traditional application, not an agent. In many ways, the creation of agents with sophisticated instruction files is a type of meta-programming that never coalesces into a deterministic form. While we could use vibe-coding to generate an application, creating an instruction file for an agent has a similar outcome, except designers never get the chance to validate the plan for each agent execution. We might restore some of that validation through a human-in-the-loop workflow, but the agent designer won’t be in the loop unless they are also the user.

Furthermore, when input comes from another agent, the expectations on input are not very clear. It’s not as unclear as coming from an untrusted user, but since the user of the calling agent might be less than fully trusted, we have to consider the possibility that a malicious user could cause the calling agent to pass dangerous inputs to the called agent. Depending on the design, that might be difficult, but proving it’s impossible is a high bar without some deterministic system in the path.

Agents Building Applications

At the apex of the framework we have agents building applications. Instead of a human using an AI tool to write code, an autonomous agent—or a multi-agent framework—is given the broad scope to architect, write, test, and deploy entire applications. Operating within bleeding-edge, emerging ecosystems like Steve Yegge’s concept of Gas Town, an overarching agent might autonomously spawn specialized “code worker” sub-agents to solve specific architectural problems. This introduces the reality of deep recursion: AI systems dynamically writing, testing, and deploying new AI systems at machine speed.

Agents Building Agents

An alternate apex is agents building other agents. While both scenarios rely on deep recursion where at least one level is indeterminate, the agents-building-agents path stores its recursive plans in natural language, rather than a programming language. Tools like Claude Cowork and Claude Code are bordering on this. Technically, they’ve always been capable of it, as a developer can create recursion somewhat trivially.

The barrier here has generally been security. It’s rather easy to say, “Agent A calls Agent B to create Agent C, which can call Agent B” (look, I just did it!). The hard part is whether that’s a good idea. Projects like OpenClaw push this further. When agents build other agent skills or update through tools like Moltbook, they are acting at this highly complex, deeply recursive layer. OpenClaw has some security controls, but not enough to prevent many users from making significant mistakes.

Another example here that illustrates the movement from applications to agents, is Gas Town in Gas City. Gas Town, the original multi-agent orchestration system for Claude Code, GitHub Copilot, and other AI agents, was an application. When Yegge wrote Gas City, a “orchestration-builder SDK for multi-agent systems”, Gas Town became “code free”, and instead a bundle of prompts and skills.

Ripple Effects: Cybersecurity

One constant that comes from more complex systems is greater challenges at securing them. All else being equal, indeterministic systems, either fully or wholly, are more complicated than fully deterministic ones. Security is traditionally about protecting deterministic plans from indeterminate actors (human hackers). Each layer adds complexity as well. Attackers will have new tools and will use them against the highest value targets that have weak points.

Fortunately, all this does not come without some benefits, both inside and outside the realm of security. Inside the realm of security, access to dynamic systems like agents allow for faster responses. As I’ve explored previously, this means security can’t wait—it will force not just an overdue commitment to defense, but complete organizational changes.

The highest value targets know this. They will rapidly adopt new defensive techniques, patching weak points faster than ever before. Where we will see more successful attacks is against more moderate value targets. Some operate efficiently and will adapt, but others will face an “adapt or fail” pressure cooker. They will suddenly find themselves defending against highly sophisticated, indeterminate automated attacks.

Overall, however, this is a narrative of optimism. As Dario Amodei notes, “the offense-defense balance may be more tractable in cyber, where there is at least some hope that defense could keep up with (and even ideally outpace) AI attack if we invest in it properly.” Security must simply shift toward robust bounding and sandboxing of environments, rather than assuming the predictability of the software operating within them.

Automation and Workflow Change

This shift in control—from human-driven applications to autonomous, recursive agents—isn’t happening just for the sake of technological novelty. Ultimately, the goal is, and has always been, automation. AI changes opportunities for automation by lowering automation costs that were previously prohibitive.

Classification and ML

Machine Learning (ML) is an AI technique that achieved broad use earlier than Generative AI. Classification and prediction tasks were the core use cases. It’s generally less well known than Generative AI because those use cases fit into embedded AI workflows that have less direct user interaction. But that doesn’t mean they haven’t been effective. Generative AI has some overlap, but it’s useful to note ML is not obsolete—it will continue to dominate specific classification tasks where the trade-offs favor highly optimized, low-compute execution.

But GenAI is shifting the math for automation’s long tail where engineering effort is the limiting factor. Traditional ML models require a significant engineering investment to train. While that engineering could theoretically be automated, doing so would bring you back to using GenAI to generate the code. When a general-purpose GenAI model can perform a task at equal quality without that upfront engineering time, it opens up a new option for countless use cases that were never practical to tackle with traditional ML. AI provides the structure to finally capture and automate the tacit knowledge we previously had to rely on humans to execute.

Workflow Change

This, and the other uses of Generative AI, allows a deeper decomposition of workflows. In prior methodologies, it was too expensive to capture the output of specific, granular steps. Those steps were done “in the head” of human workers, existing only as “tacit knowledge.” A workflow that may have produced better results might have been avoided because the human cost of data preparation or classification was too high.

While it’s possible to replicate the same workflows, trade-offs have shifted. Consider a nurse who has learned a new symptom for a patient. That nurse may lack the depth of medical knowledge or the patient’s full history, so may not be able to do more than record that information until the patient’s doctor can review it. But an AI system can reanalyze a patient’s information nearly instantly. It can recategorize data, make new recommendations, or provide the nurse with the relevant medical information and patient history. This could allow next steps that both improve efficiency, but also improve outcomes. Maybe it’s an extra test, or an extra question, or a life-saving reaction.

Because AI lowers the cost of executing small, indeterminate tasks, we can now decompose workflows further. What workflow is optimal depends heavily on hand-off costs. In the example of the nurse, the hand-off costs to the doctor were the impediment. Human to human or human to machine hand-offs are expensive compared to machine-to-machine. When tasks shift from human dependent to machine dependent, a reorganization of workflow makes sense. A particular flow which was used to avoid handoffs may no longer be necessary. More importantly, those handoffs that remain, will have higher relevance than before, and optimizing for them, rather than those that are no longer needed, takes priority.

Initially, we should expect to see pilots, trials, and first iterations operate within existing workflows. Changing workflows requires planning, training, and is hard to reverse or do incrementally. As such, it follows in later iterations. But many of the largest gains are realized with those iterations.

The Human Side

Workflow change can also have a significant impact on satisfaction amongst workers. Hand-offs can be the most frustrating type of work, depending on your personality type. Human to machine hand-offs become frustrating when flexibility is lacking, and you feel like your task is to fit a round-peg into a square hole. Human to human hand-offs can sometimes be enriched by the personal interaction, but they also expose you to misaligned goals, competing priorities and personality conflicts. Personal interactions are a lot more reliably fun when you get to pick the individuals and circumstances.

Worst of all, machine to human hand-offs can create the impression that you’re serving the machine, not the inverse. All hand-offs can have that effect somewhat, but it’s especially hard if there’s an endless list of machine generated work. It helps to detach from the “end” and focus on the progress here, but when an organization turns that into a metric, ruthlessly gamifies it, and fails to consider the impacts, it requires extreme stoicism to avoid burnout.

It’s important to remember the human side with workflow change. Ruthless metrics fail in the long term. Leaders should watch for that, and avoid allowing short-term goals to overwhelm long-term health. It’s not always clear that this fulfills the “bottom-line” if that’s financial performance. It’s always clear it’s better for broader goals than financial, but even for financial goals, the benefits are likely there, though harder to see.

Conclusion

The era of software as purely rigid, deterministic planning is ending. In its place is the rapidly expanding Agent Ecosystem. By integrating indeterministic models into our autonomous systems, and allowing agents to build other agents, we are trading perfect predictability for unprecedented scale and capability.

As the scope of these systems increases, our primary job shifts from writing static instructions to managing boundaries. We must build robust technical sandboxes to protect our cybersecurity, and we must build equally robust organizational boundaries to protect human workers from the burnout of endless machine-to-human hand-offs. We have to design systems that serve humans, not the other way around.

The question is no longer just “What can the chatbot say?” The real questions are: How much scope are we willing to give to indeterminate plans? How will we effectively bound the recursive systems of tomorrow? And, ultimately, how do we bound ourselves?

1. There is debate on what the minimum requirement is to be an “agent”. For this framework, we’ll use the looser form that does not require continual autonomy, but simply the ability to create and execute a plan, which may still involve supervision. Just two of many models of describing agency: arXiv (Huang et al., May 2024); arXiv (Feng et al., June/July 2025)

2. AI chatbot providers may give users a way to define deterministic workflows, but you can think of these as user-built tools; a very simple version of application building.