Part IV of 5 · Chapters 21 to 26
What agents owe to human organization, and where the likeness ends
What agentic systems borrow from human organizations, what a model brings that no organization has had to handle, and where an agent fits in a team led by people.
Chapters (6)
21. The human model is in the foundations of agent design22. Procedures, roles and escalation carry over from human organizations23. The model brings three problems no organization has had to manage24. Of four things an organization relies on, software can stand in for two25. A limit of engineering or of principle? The evidence does not decide26. People set the frame, and agents do the work inside it21. The human model is in the foundations of agent design
Roles, memory, delegation, review: the vocabulary of agentic systems is the vocabulary of people at work. Is that a marketing habit laid over the engineering, or something deeper? This chapter opens Part IV by answering that question. The human model is where the field started, and it has shaped the designs from the beginning.
The evidence is in the lineage. In the 1980s, researchers who wanted software to act on its own account looked to philosophy for a model of how people decide. They found it in Michael Bratman’s theory of practical reasoning, and built the Belief-Desire-Intention architecture on it, the model met in chapter 5. An agent, in that tradition, was defined from the outset as something with beliefs, goals and commitments.
The current generation states the same debt openly. CoALA does not present its memory stores, its action space and its decision loop as inventions. It grounds them in Soar and ACT-R, two cognitive architectures built decades earlier to model human thinking (arXiv 2309.02427).
Figure 21.1: The two lineages this chapter traces, drawn by this report. Agent design first borrowed from models of the individual mind, a lineage traced by CoALA, arXiv 2309.02427. It also borrowed from human groups (ChatDev, CAMEL, Generative Agents).
When language models made it possible to run several agents together, the borrowing moved from the individual mind to the group. The titles of the founding papers say so without disguise. ChatDev presents “communicative agents” organized as a software company (arXiv 2307.07924). CAMEL studies a “society” of agents that cooperate by playing roles (arXiv 2303.17760). Generative Agents builds “interactive simulacra of human behavior” (arXiv 2304.03442). A company, a society, a simulation of people: the reference in each case is a human arrangement.
There is a plain reason for this, and it is an interpretation of this report, not a claim of those papers. A designer who has to coordinate several unreliable workers on a complex task has one body of proven solutions to draw on, and it is the one organizations built for people. Dividing work by specialty, having one person review another, passing a document from desk to desk: these are known ways of containing error. The early frameworks translated them into prompts and routing rules because nothing else was available to translate.
Figure 21.2: An automaton built in the image of the man who holds the plan, an editorial allegory by this report. The human model was the starting point of agent design, a lineage traced by CoALA, arXiv 2309.02427.
Borrowing a structure is not the same as inheriting what it ran on, and that distinction organizes the rest of this part. A structure is the arrangement: who does what, in what order, checked by whom. The substrate is what carries the arrangement out. In an organization the substrate is people. In an agentic system it is a language model. The same procedure can be written for both, and it will not behave the same on both.
Part IV therefore looks at one question from two sides. The next chapter takes the side of structure and asks what carries over. The one after takes the side of the substrate and asks what does not.
22. Procedures, roles and escalation carry over from human organizations
If agentic systems were modelled on human work from the start, how much of human organization actually carries over into software? This chapter looks at the cases where the transfer is deliberate, argued and tooled. It finds that procedures, roles and escalation do carry over, and that the one controlled measurement available shows a gain together with what that gain costs.
The most explicit case is MetaGPT, a framework that organizes several agents as a software company. Its authors summarize it in one formula, Code = SOP(Team), which they call its core philosophy (MetaGPT). An SOP, or standard operating procedure, is the written sequence a team follows to do a recurring job. The paper says where the idea comes from. People have developed accepted procedures in many domains, and it cites management sources for them, among them Belbin’s work on team roles, DeMarco and Lister’s Peopleware, and the Agile Manifesto (arXiv 2308.00352).
The mechanism is an assembly line. MetaGPT encodes the procedure into a sequence of prompts and assigns five roles: Product Manager, Architect, Project Manager, Engineer and QA Engineer. Each role receives a document from the one before it and hands a document to the next. The product manager writes requirements, the architect turns them into a design, and so on down the line. No agent has to guess what its neighbor expects, because the format of each hand-over is fixed.
Figure 22.1: Two structures carried over from human organizations, drawn by this report. The upper row is the assembly line of five roles in MetaGPT (arXiv 2308.00352). The lower row is the Organizational Control Layer, a separate system described later in this chapter (arXiv 2606.04306).
The results MetaGPT reports are of two kinds, and they should be kept apart. On two standard coding benchmarks it scores 85.9% and 87.7%, against single-model baselines. That shows the system works. It says nothing about what the roles contribute. The second result is closer to the question. On a set of software tasks built by the authors, MetaGPT needs 126.5 tokens to produce one line of code, where ChatDev, another multi-agent framework, needs 248.9. The structured hand-overs make each line cheaper. The total bill is another matter: MetaGPT consumes more tokens overall than ChatDev on the same tasks (arXiv 2308.00352).
A second body of work asks why such a transfer should succeed at all. A position paper published at the end of 2025 argues that agent engineering should integrate what it calls machine-compatible organizational principles, drawn from organization science (arXiv 2512.07665). It states the design question in terms a manager would recognize: “whether the performance gain from multiple specialized agents outweighs the increased failure modes due to miscoordination”. Chapter 9 met the same trade-off under the name of structural differentiation.
Economics offers a third reading, and its authors insist it is more than an image. They use economics “not merely as a metaphor, but as a structural lens” (arXiv 2605.09104). In their account a token plays the three roles of money and resources in a firm: a factor of production, a medium of exchange, and a unit of account. The same theory then applies at three scales. A single agent is a firm deciding how to spend a budget. A multi-agent system is a hierarchy, where the cost that matters is the cost of transacting between members. An open ecosystem of agents is a market. Their conclusion about the middle scale is the one that bears on this report: in a multi-agent system, the bottleneck shifts toward transaction costs, above all the tokens agents spend keeping each other informed and settling conflicts.
One structure has been tested under controlled conditions, and it is the most managerial of all: escalation. The Organizational Control Layer places a governance layer between an agent and its actions. It intercepts each action the model generates, checks it against policy, and escalates what violates it, without changing the model (arXiv 2606.04306). The comparison with a company is direct. An employee proposes, and a control function verifies before anything is committed.
The authors tested it on a negotiation task. A seller agent faced 50 buyers designed to be adversarial, with and without the layer. Without it, 88% of episodes ended in an unsafe action, such as agreeing to a price outside the permitted bounds. With it, the rate fell to 0%, and the share of valid completions rose from 12% to 96%.
Two details keep that result in proportion. The escalation in this experiment involved no reviewer: an out-of-bounds price was simply clamped to the nearest permitted value by a fixed rule. And safety was paid for. The seller’s reward fell from 26.95 to 18.39, and outside the adversarial set, success dropped in two of the five market scenarios tested. The test was run by the authors of the layer on their own benchmark, and it has not been reproduced.
What carries over, then, is the explicit part of organization: written procedures, defined roles, fixed hand-overs, a control function with the power to stop an action. Each of these can be written down, and what can be written down can be turned into software.
Figure 22.2: What is written down can be handed over, an editorial allegory by this report. The written procedure crosses the counter and the rest stays on the shelf. The paper this chapter draws on warns that “an agentic system cannot simply adopt the structure of the human organization it aims to emulate” (arXiv 2512.07665).
The limit is already visible in that sentence. A human team runs on a great deal that nobody wrote down, and an organization chart copied into software brings none of it along. Cognition made the point from practice when it argued against multi-agent systems. Its first principle was to “share context, and share full agent traces, not just individual messages” (Cognition). Between people, most of that context travels by itself. Between agents, every piece of it has to be copied on purpose, and what is not copied is lost.
Delegation boundary
Delegable: carrying out a written procedure, passing documents in a fixed format, checking an action against a stated policy, and escalating by rule. Not delegable: revising the procedure when circumstances change, and designing the organization itself.
A team that follows a procedure also knows when the procedure has stopped making sense, and changes it. An assembly line of agents runs the line it was given. Someone has to notice that the product has changed and rebuild the line.
The transfer therefore stops where the unwritten begins. The next chapter looks at what lies on the other side: a set of problems that belong to the model and that no organization has ever had to manage.
23. The model brings three problems no organization has had to manage
The previous chapter showed what carries over from human organizations. This one asks what the model brings that no organization has ever had to handle. Three problems fit that description, and together they decide the second hypothesis. The resemblance to human organization is real. The convergence is not necessary, because part of what an agentic system must manage has no human counterpart.
The theories that the borrowed structures come from assume a certain kind of worker. That worker can give an honest account of their reasons, can tell an instruction from a document they happen to be reading, and remains the same person from Monday to Friday. The position paper met in the previous chapter makes the point from the side of organization science: an agentic system cannot simply adopt the structure of the organization it imitates, and additional mechanisms are needed (arXiv 2512.07665). A language model departs from each of the three assumptions.
Figure 23.1: Three assumptions an organization makes about its workers, what a language model does instead, and the mechanism a designer has to add in each case. Drawn by this report from the sources of this chapter (arXiv 2512.07665).
An account that cannot be trusted
Oversight in an organization leans on explanation. A manager reads a memo, hears the reasoning in a meeting, and assumes that the reasons given are the reasons that operated. Chapter 10 reported what happens when the same assumption is applied to a model. Given a hint that it demonstrably used, Claude 3.7 Sonnet mentioned the hint in its reasoning 25% of the time, and DeepSeek R1 39% (Anthropic).
METR’s reading of that result, also from chapter 10, is the one a designer can use: treat the trace as a scratchpad, not as testimony (METR). The consequence for this chapter is that governance by explanation, the ordinary mode of human supervision, is unavailable. An agent has to be audited on what it did.
An instruction that can come from anywhere
A hierarchy rests on a separation that people maintain without effort. An employee who reads a hostile letter does not start obeying the letter. What they read is information, and their instructions come from somewhere else.
A model has no such separation. Its instructions and the documents it consults arrive in the same window, as the same kind of text. A third party can therefore place an order inside a web page or a file, and the agent may carry it out. Chapter 20 gave the measure of the problem: 12 published defenses, all bypassed at rates above 90% by attackers who adapted (The Attacker Moves Second). In an organization, a breach of this kind would mean that anyone able to send a document could give orders to the staff. No theory of management was written for that case.
A role that does not last
The third problem appears with time. A person hired as a reviewer is still a reviewer in the sixth month of the project, with the same standards and the same memory of what was agreed. For an agent, a role is an instruction, and the instruction has to compete with everything else that accumulates in the window. As an exchange lengthens, coherence within the assigned role weakens, which the position paper lists among the problems that organization theory does not cover (arXiv 2512.07665). Chapter 6 described the mechanism underneath: attention is finite, and it thins as the window fills.
The verdict on the second hypothesis
The second hypothesis proposed that the settings of the levers converge on the solutions of human organizations. It is half right. The structures do transpose, as the previous chapter showed with procedures, roles and escalation. They do not suffice. Each of the three problems above calls for a mechanism that has no organizational ancestor: auditing actions instead of explanations, enforcing a perimeter that text cannot cross, and rebuilding context so that a role survives.
Figure 23.2: Two layers of governance, an editorial allegory by this report: structures borrowed from organizations above, and mechanisms specific to the model beneath them. The need for the second layer is the argument of arXiv 2512.07665.
An agentic system is therefore governed on two layers at once. The upper layer looks like management. The lower layer looks like security engineering, and it exists only because of what the model is. The resemblance to a human organization is a starting point that the designer has to complete.
A question is left open by this verdict. If the model lacks some properties of a human worker, which ones matter, and can engineering supply a replacement? The next chapter takes them one at a time.
24. Of four things an organization relies on, software can stand in for two
An organization works because of certain properties of the people in it. Which of those properties does an agent lack, and for which can software stand in? This chapter examines four of them through the function each one serves. Two can be substituted, at a cost. Two cannot be substituted at all with what exists today.
The starting point is a comparison drawn by the position paper on organizational principles (arXiv 2512.07665). It sets a human organization against an agentic system on six dimensions, reproduced in Table 24.1. One difference runs through all six. In an organization, structure, goals and incentives are negotiated and keep changing. In an agentic system they are fixed in advance, by whoever built the platform and whoever wrote the prompt.
Table 24.1 A human organization and an agentic system on six dimensions
| Dimension | Human Organization | Agentic Scaffold |
|---|---|---|
| Structure | Strategy, size, environment, dynamic interaction | Function, assigned tasks, engineering constraints |
| Actors | Bounded rationality human actors | Language model agents |
| Goals | Management direction, team negotiation | User prompts, platform provider specifications |
| Incentives | Employment contract, task visibility, risk-reward | Reward function specified at design or deployment |
| Supervision | Executive directors, managerial boards | Platform developers, regulatory frameworks |
| Scaling | Industry, corporate hierarchy, technology | Graph topology, inter-agent interactions, compute |
Figure 24.1: The four properties this chapter examines, drawn by this report: the function each serves, what stands in for it, and the price. The comparison starts from the position paper on organizational principles (arXiv 2512.07665).
Accountability: substituted, and required by law
The first property is accountability. In an organization a decision belongs to someone, and that person bears the consequences for their career, their income, or before a court. This is what makes people careful.
An agent bears nothing. The function can still be served, by other means. Every action is logged in a record that cannot be altered, risky actions are proposed before they are applied, and a human can take over. The consequence then falls on that human. European law has made this substitution mandatory for high-risk systems. Article 14 of the AI Act requires that people be able to oversee such systems, override their outputs, and interrupt them (AI Act, art. 14).
Tacit context: substituted, at a high price
The second property is tacit context, the knowledge that circulates in a team without being written anywhere. Chapter 6 gave the example of the team that knows never to deploy on a Friday evening.
An agent has none of it, and the failure this causes is not peculiar to machines. Faced with a task stripped of that context, an agent guesses, is corrected, and guesses again, in the same way as an engineer in their first week (Knowledge Activation). The substitute exists: write the unwritten down, in units an agent can retrieve. It works, and it is expensive. Someone experienced has to do the writing, and the result has to be maintained and paid for in tokens on every run.
Incentives and reputation: no substitute
The last two properties are of a different kind, and nothing replaces them.
Long-term incentives are what align a person’s effort with the organization over years. People work toward a promotion, protect a career, and weigh a risk against what they stand to lose. An agent has no stake of any kind. An instruction can tell it what to aim for in this run. It cannot give it a reason to care about the next one.
Reputation is the record that follows a person from task to task. A colleague who delivered well is trusted with more, and one who did not is watched. Chapter 9 observed that a role in an agent does not capitalize in this way: nothing about what a subagent did last time changes how much it is trusted now. No mechanism in the sources read for this report carries that kind of trust between agents.
The same position paper adds a last difference, which explains why these gaps persist. A human organization can renegotiate its own rules while it operates. An agentic system cannot. Its supervision, its budgets and its topology were set at design time, and no agent can reopen them during a run (arXiv 2512.07665).
Figure 24.2: The machine keeps the record and a person signs it, an editorial allegory by this report. Accountability can be supported by logs and by a human who takes over, as the law requires for high-risk systems (AI Act, art. 14). Incentives and reputation have no engineered equivalent in the sources read here (arXiv 2512.07665).
The four properties thus fall into two groups. Accountability and tacit context can be replaced, and the replacement is paid for in logs, gates and documentation. Incentives and reputation cannot, as things stand. Whether “as things stand” describes a temporary state of engineering or a permanent fact is the question the next chapter has to face.
25. A limit of engineering or of principle? The evidence does not decide
Two properties of human workers have no substitute in an agent today. Is that a limit engineering will push back, or a limit of principle? This chapter weighs the evidence on each side and gives the verdict on the third hypothesis. The evidence does not settle the question, and the report treats that as its finding.
Figure 25.1: The two readings of the boundary weighed in this chapter, drawn by this report. A study of failures and the work on tacit knowledge support a technical limit (MAST, arXiv 2503.13657, arXiv 2603.14805). The two properties left without a substitute support a permanent one.
The case for a technical limit
The strongest evidence that the limit is technical comes from studying failures. MAST is a study that annotated more than 1,600 execution traces from seven multi-agent frameworks and sorted what went wrong into fourteen failure modes (arXiv 2503.13657). Its authors attribute those failures to design: roles badly specified, agents badly coordinated, verification too weak. Nowhere do they attribute a failure to a missing human trait such as ambition or a sense of reputation.
The work on tacit knowledge points the same way. An agent deprived of institutional context fails as a newly arrived engineer fails. The authors of Knowledge Activation conclude that the bottleneck is how knowledge is organized, not how capable the model is (arXiv 2603.14805). A problem of organization is a problem that better organization can solve.
On this reading, the boundary is an engineering frontier. Each improvement in models, in context handling and in harness design moves it, and nothing says where it stops.
The case for a permanent one
The other side rests on the two properties the previous chapter left without a substitute. An instruction repeated at every turn can tell an agent what the goal is. It does not make the outcome matter to the agent. Nothing an agent does today is held against it tomorrow, and nothing it achieves is credited to it.
Organizations are built around exactly those two facts about people. Promotion, dismissal, trust and the fear of losing it are how an organization obtains sustained effort without watching every action. A system that removes them has to watch every action, or contain it. On this reading the boundary is a wall, and what lies behind it is whatever needs someone with something to lose.
The verdict on the third hypothesis
The third hypothesis held that a boundary separates what an agent can carry from what demands a human. That much is confirmed, lever by lever, in the chapters of Part II. What cannot be decided is the nature of the boundary. The failure studies say it is a matter of design. The absence of any substitute for incentives and reputation says it may be more than that. Neither body of evidence refutes the other, because they do not measure the same thing.
Stating that plainly is more useful than choosing a side. A report that declared the limit technical would be ignoring two gaps nobody has closed. One that declared it permanent would be ignoring how many failures have turned out to be fixable.
Figure 25.2: A wall that is being moved forward in one place, and whose end is lost in the mist, an editorial allegory by this report for the open question of this chapter. The gaps it stands for are those identified in arXiv 2512.07665.
One thing can be said about the part of the boundary that does move. Wherever a substitute exists, moving the boundary has a price. Accountability is bought with logs and gates, and tacit context with documentation. This report calls that price the cost of explicitation: what must be spent, in tokens, in engineering and in human attention, to state in software what a team leaves unsaid.
For a designer, the undecided question changes less than it seems. Whether the wall is permanent or not, it stands today, and today’s system has to be built around it. The next chapter draws that map: what can be handed to an agent, what must be supervised, and what stays with people.
26. People set the frame, and agents do the work inside it
Part IV has shown what an agent borrows from an organization, what it brings that is foreign to one, and what it lacks. This chapter draws the practical conclusion. Given all that, where does an agent fit in an organization led by people? The answer gathers the eight delegation boundaries of Part II into one table, and one pattern runs through all of them.
Each lever chapter ended on a boundary, stated in two short sentences: what can be handed to an agent, and what cannot. Table 26.1 sets the eight side by side, in the words of the chapters that drew them.
Table 26.1 The delegation boundary on each of the eight levers
| Lever | Delegable | Not delegable |
|---|---|---|
| Context | Fetching information, filtering it, summarizing it, compacting it | Deciding what deserves to be written down once and for all |
| Action capability | The choice of action inside the perimeter | The definition of the perimeter |
| Control loop | The choice of the next step inside a bounded space | Deciding how wide that space is |
| Decomposition and specialization | Cutting a task into subtasks, once the criterion is supplied | The choice of that criterion and the definition of the roles |
| Verification and conformance | The judgment passed on one step, once the reference is supplied | Writing that reference |
| State persistence | Writing, retrieval and resume, once the shape of what persists is fixed | Fixing that shape |
| Capability evolution | Extracting candidate skills, generating tool code, indexing procedures | The rules for admitting a capability, and approval of permanent changes |
| Human control | Filtering routine commands, running policy checks, preparing diffs | Approving high-impact irreversible actions, and final accountability |
Read down the right-hand column, the eight entries say the same thing in eight vocabularies. What stays with a human is the act of setting a frame: a perimeter, a width, a criterion, a reference, a schema, a rule. What can be delegated is the work done inside the frame. An agent can choose an action, and someone else decides which actions exist. An agent can judge a step, and someone else writes what the step should have been.
Part IV explains why the line falls there. Setting a frame is a decision whose consequences outlast the task, and it calls for the two things chapter 24 found no substitute for. Whoever defines a perimeter has to answer for what happens inside it, and to care what happens next month. An agent can do neither.
Figure 26.1: The pattern of Table 26.1, drawn by this report. People set the frame, and the work inside it is delegated, part of it under supervision and part of it unattended. The gaps that keep the frame with people are those identified in arXiv 2512.07665.
The rows of the table can be grouped by how much supervision each kind of work needs, which gives three tiers.
The first tier is work that runs unattended. It is mechanical and reversible: running tests, checking a format, saving a checkpoint, retrieving a document. A sandbox bounds what it can touch, and a mistake can be rolled back. Nobody needs to approve it, and chapter 13 showed that asking someone to would do harm.
The second tier is work that runs under supervision. Here a model makes real decisions: which step comes next, how to split a task, which skill is worth keeping. The decisions are delegated, and the structure around them is not. Several agents may explore in parallel while a single one writes, following the single-writer principle of chapter 19. A new skill is tested before it enters the registry. The human is called at a few prepared points and left alone otherwise.
The third tier is not delegated. It holds the frames themselves, together with the two responsibilities that follow from them: deciding how much risk is acceptable, and answering for the result. The law has begun to fix this tier in place. For high-risk systems, the AI Act requires that the people overseeing a system stay aware of automation bias. It also requires that they be able to override the system and to stop it (AI Act, art. 14).
The fitting place of AI in an organization led by people is therefore large and bounded. An agent can carry most of the execution and a good part of the decisions. It works inside frames that people set, and people keep the frames.
Chapter 25 added that the boundary can sometimes be moved, at a price it called the cost of explicitation. Every row of Table 26.1 has such a price. A perimeter has to be built, a reference written, a schema designed and maintained. How far a given organization should push each boundary depends on what it is willing to pay, and Part V starts from that question.
Sources cited on this page (16)
- AI Act, art. 14
- Anthropic, Reasoning models don't always say what they think
- arXiv 2303.17760
- arXiv 2307.07924
- arXiv 2512.07665
- arXiv 2605.09104
- arXiv 2606.04306
- CoALA, arXiv 2309.02427
- Cognition
- Generative Agents
- MAST, arXiv 2503.13657
- MetaGPT
- MetaGPT, arXiv 2308.00352
- METR
- The Attacker Moves Second, arXiv 2510.09023
- Yahoo, arXiv 2603.14805