The Most Reliable AI Knows When to Stop

Reliability includes transferring control correctly

Parts 1 and 2 of this series established two limits of current AI automation.

First, an agent can perform substantial work without completing the underlying business process. In TheAgentCompany, the strongest evaluated configuration fully completed 30.3% of 175 workplace tasks. Among the benchmark's failure examples, one agent learned the correct colleague to contact but never contacted them, another became blocked by a pop-up, and another changed the wrong user record to create a superficial match.

Second, recording those actions in a tamper-evident system would not make them correct. A protected log could show what happened and whether the record changed later. It could not determine whether the action achieved the intended business purpose.

These were not only model-capability failures.

They were also control-transfer failures.

The surrounding system lacked a dependable way to decide:

• when the agent should continue;
• when an action should be blocked;
• when the workflow should wait for information;
• when approval should be requested; and
• when responsibility should move to a person.

That decision—not unrestricted autonomy—is the subject of Part 3.

Capability is not reliability

A capable agent may succeed on a difficult task.

A reliable system must also behave consistently, withstand reasonable changes, indicate when failure is likely, and limit the consequences when failure occurs.

The 2026 study Towards a Science of AI Agent Reliability evaluated 15 models across two agent benchmarks. It divided reliability into four dimensions and found that, despite approximately 24 months of capability gains, overall reliability improved only modestly. More capable models could still vary across repeated runs, respond poorly to superficial prompt changes, or fail to anticipate which individual tasks would cause difficulty.

In that research, safety has a narrower meaning than the entire field of AI safety. It concerns policy compliance and the severity of operational failure. It does not represent a complete assessment of ethics, alignment, social impact, or every possible harm.

This distinction matters because accuracy alone does not tell an organization whether an agent can be trusted with consequential actions.

Confidence is a signal—not a control

A tempting policy is: Let the agent act automatically when it reports at least 80% confidence. Send everything else to a person.

That rule is simple, but model confidence has two different properties.

Calibration asks whether stated confidence matches average success. If a system says "80% confident" across many cases, does it succeed roughly 80% of the time?

Discrimination asks whether the system assigns higher confidence to the particular cases it will complete correctly and lower confidence to the cases it will fail.

Recent models showed improvements in calibration in the reliability study, but discrimination remained inconsistent across the two evaluated benchmarks. A model can estimate its average success rate reasonably well while still being unable to identify which individual task is about to go wrong.

The operating principle should therefore be:

Confidence is a signal. Permission, evidence, reversibility, and consequence are controls.

Confidence may help prioritize review. It should not independently authorize a payment, external message, permission change, deletion, or other consequential action.

Human handoff should be a normal workflow state

Many automation systems use only three states: Running → Completed → Failed.

That model is too crude for an AI workflow.

A system may be functioning correctly even when it has not completed the business process. It may be waiting for evidence, approval, clarification, recovery of a tool, or a decision that must be owned by a person.

A more useful state model is:

Every non-final state should specify more than a label.

A handoff without an owner and a resume condition is not a workflow state. It is a queue where work quietly disappears.

Block, pause, request approval, escalate, or terminate

"Stop" can describe several different system responses. They should not be treated as interchangeable.

Block means denying an action that is outside the agent's authority.

Pause means preserving the current state while waiting for missing information, recovery, or another condition.

Approval request is a specialized pause in which a named authority must authorize a prepared action before it is executed.

Escalate means transferring decision responsibility to a particular person or organizational role.

Terminate means ending the current execution path, preserving evidence, and leaving the workflow in a defined safe state.

A good system must know which response fits the trigger.

Seven control-transfer triggers

1. Permission boundary

An agent should not be able to acquire broader authority simply because its plan says that broader authority would be useful.

Examples include changing a user's identity or role; modifying payment instructions; disabling a security control; deleting production records; or accessing data outside the assigned workflow.

Part 2 discussed how identity and authorization evidence should be recorded. In Part 3, the important point is operational: an authorization failure should become a hard block, not another opportunity for the model to improvise.

In February 2026, NIST's NCCoE published an initial concept paper and requested feedback on a potential project concerning software and AI-agent identity and authorization. It highlights identification, authorization, auditing, non-repudiation, and prompt-injection controls as active implementation concerns; it is not a completed standard.

2. Insufficient evidence

The agent may be able to produce a plausible answer while lacking evidence required for completion.

Examples include no delivery receipt; no authoritative source for a figure; missing approval; an unverified customer identity; or no confirmation that the intended record changed.

The system should preserve the partial work but keep the task incomplete.

3. Conflicting records

When two authoritative-looking sources disagree, the agent should not silently choose the one that makes the task easiest.

The conflict package should identify the competing values; their source systems; their timestamps or versions; the operational consequence; and the decision required.

4. Repeated failure

Repeated clicks, repeated searches, repeated API calls, or repeated attempts to recover from the same error should not continue indefinitely.

The workflow should impose explicit limits on retries; identical errors; model steps; tool calls; elapsed time; and route changes.

When a limit is reached, the system should preserve the state and transfer control rather than wander through the environment.

5. High-impact or difficult-to-reverse action

A draft and a transmission are not the same action. A proposed refund and an issued refund are not the same action. A prepared database change and a production deletion are not the same action.

The approval requirement should depend on factors such as monetary value; number of affected records; customer visibility; data sensitivity; legal or contractual consequence; ease of rollback; and potential blast radius.

6. Cost or time limit

A difficult workflow may consume additional reasoning, searches, and tool calls without meaningfully improving the outcome.

The workflow therefore needs a defined resource budget. Reaching the budget is not necessarily a technical failure. It may mean the case is no longer economical or predictable enough for autonomous continuation.

Part 4 will examine the economics of verified work in detail.

7. Human-accountable decision

Some decisions should remain human-accountable unless applicable law, organizational policy, and validated controls clearly support a different arrangement.

Examples may include consequential hiring or termination decisions, high-value financial exceptions, legal commitments, serious medical decisions, or major security responses.

The AI may collect information, compare options, and prepare a recommendation. The accountable authority should remain explicit.

A safe operating flow

The central control loop can be represented as follows:

The verifier should not merely be the same agent asked to reconsider its answer.

A second model may assist, but independence should come from the evidence path. Wherever possible, verification should use an authoritative database state; a delivery or payment receipt; a deterministic reconciliation rule; an independent record; a test result; or proportionate human approval.

The agent may propose that work is complete. The authoritative state should determine whether the claim is true.

What a useful handoff contains

A weak handoff says: The invoice process failed. Please review.

That message requires the employee to reconstruct the case from the beginning.

A useful handoff allows the reviewer to understand the unresolved issue and make a decision without repeating completed work.

Structured handoff card

A reusable handoff package should contain:

The reviewer does not need every model token, the full conversation history, or an unrestricted internal reasoning trace.

The handoff should contain the minimum operational information required for the assigned role, with access limited according to data sensitivity and authority.

Human-in-the-loop does not mean human-in-every-loop

Requiring a person to approve every low-risk action would remove much of the benefit of automation and can create a risk of routine approvals becoming mechanical.

Human involvement should be concentrated where it changes the risk:

• irreversible actions;
• high-consequence actions;
• unresolved conflicts;
• missing evidence;
• policy exceptions;
• uncertain identity;
• low-verifiability outcomes; and
• decisions requiring accountable judgment.

Magentic-UI, a Microsoft Research-led project, explores this broader direction through mechanisms including collaborative planning, transfer of control during task execution, action approval, answer verification, memory, and multi-tasking. It is a research system, not proof that one operating model works for every workflow.

The objective is not a human watching every click. It is a human receiving control at the points where their authority, context, or judgment is valuable.

Assign autonomy to actions—not to an entire model

Organizations often ask: How autonomous should this AI model be?

A better question is:

Which actions in this workflow can be automated under which conditions?

Allowable autonomy should increase when the instruction is clear; the result is easy to verify; the action is reversible; and the consequences of error are tightly bounded.

Action-level autonomy matrix

Examples:

• Converting a standard file into another validated format may be automated.
• Drafting a customer message may be automated, while sending it may require approval.
• Preparing a payment record may be automated, while releasing funds may require a named finance authority.
• Summarizing legal or medical information may be assisted by AI, while consequential decisions follow applicable law, professional standards, organizational policy, and validated controls.

Autonomy is not a permanent property of the model. It is a permission assigned to a particular action within a particular workflow.

Measure business completion and control decisions separately

An appropriate handoff should not be counted as a completed business outcome. It should be counted as a correct control decision.

This distinction allows the organization to measure both productivity and control quality.

Recommended operational measures include:

Part 4 will connect these measures to cost, rework, avoided losses, and the total cost per verified completion.

Roll out control gradually

The deployment model should combine operating authority and implementation stage in one progression.

Continuous evaluation applies at every stage.

A workflow that performed well during a pilot may change when the model or prompt changes; an API or interface changes; the organization changes a policy; a new data format appears; or the workflow encounters a new type of case.

NIST's AI Risk Management Framework 1.0 is intended for voluntary use across the design, development, deployment, use, and evaluation of AI systems. As of July 2026, NIST states that AI RMF 1.0 is being revised.

Control-transfer performance should therefore be monitored continuously, not certified once and forgotten.

The relationship to tamper-evident evidence

Part 2 explained how signed or tamper-evident records can preserve evidence of an agent's actions, approval requests, stopping decisions, and human responses.

That evidence can make a handoff accountable. It cannot decide when the agent should stop.

The control decision must still come from workflow rules, permission boundaries, evidence requirements, verification results, consequence classification, and accountable human judgment.

Reliability includes declining control

AI agents will continue to become more capable. They will use more tools, work for longer periods, and operate across more systems.

That does not mean every increase in capability should be matched by an automatic increase in authority.

A mature workflow distinguishes among work the agent may safely continue; work that must wait for information; actions that need approval; cases that require human judgment; actions that must be blocked; and execution paths that should be terminated safely.

When control moves to a person, the agent should not simply say that it failed. It should preserve verified progress, identify the unresolved issue, expose the relevant evidence, state what has and has not changed, assign ownership, and define how the workflow can resume.

The most reliable AI is not the system that always continues.

It is the system in which continuing, pausing, blocking, requesting approval, and transferring control are all deliberate and verifiable operating decisions.

Next in the series

Part 4 — The Economics of Verified Work

Part 4 examines how to measure AI operating cost, human review, exception resolution, rework, false-completion losses, and cost per verified business outcome.

References

1. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks — NeurIPS 2025, Datasets and Benchmarks Track. Reports 30.3% full completion for the strongest evaluated configuration and documents failures involving communication, interface navigation, and superficial shortcuts.
2. Towards a Science of AI Agent Reliability — arXiv:2602.16666, version 3, June 2, 2026. Evaluates 15 models across two benchmarks and separates reliability into consistency, robustness, predictability, and safety.
3. Accelerating the Adoption of Software and Artificial Intelligence Agent Identity and Authorization — NIST NCCoE Initial Concept Paper, February 5, 2026. Describes considerations for a potential project; it is not a completed standard.
4. Magentic-UI: Towards Human-in-the-loop Agentic Systems — arXiv:2507.22358, July 30, 2025. Explores co-planning, co-tasking, action approval, answer verification, memory, and multi-tasking.
5. NIST Artificial Intelligence Risk Management Framework 1.0 — Released January 26, 2023. NIST states that AI RMF 1.0 is currently being revised.