The Hardest Part of AI Automation Is Finishing the Work

The most convincing moment in an AI demonstration is often the moment just before the real work begins.

A model reads an instruction, searches for information, produces a polished analysis, drafts an email, or writes a piece of code. The result appears in seconds, and the demonstration feels complete.

But a business process does not end when an answer appears.

The information must come from the correct source. The intended system must actually be updated. A document may need to be stored in a specific location, sent to the correct person, and approved by someone with the appropriate authority.

Finally, the organization must confirm that the expected business state was actually reached.

Producing a plausible output and completing the underlying business process are not the same thing.

That gap is one of the central problems in applied AI.

Evaluating the whole job—not merely the answer

TheAgentCompany is a research benchmark designed to evaluate AI agents inside a simulated software company.

Rather than asking isolated questions, it assigns agents 175 workplace tasks requiring them to interact with several systems. The environment includes GitLab for code and internal documentation, Plane for project management, Rocket.Chat for communication, ownCloud for documents, and a local workspace containing a browser, code editor, Python environment, and Linux terminal.

The tasks cover software engineering, project management, data science, administration, human resources, finance, and related work. Some require the agent to communicate with simulated colleagues to obtain information that is not present in the original instruction.

Each task is divided into checkpoints. Depending on the workflow, the evaluation may check whether the agent:

• retrieved the correct information;
• completed the required calculation;
• created the specified file;
• updated the intended system;
• contacted the correct colleague; and
• submitted or shared the final result.

A task counts as fully completed only when all required conditions are satisfied.

This distinction matters because useful progress is not the same as a completed business outcome.

An agent may find a document, extract the right numbers, and draft a report—yet still fail to save the file, send it to the right person, or complete the final approval step.

The 2025 result: 30.3% full completion

In the NeurIPS 2025 version of TheAgentCompany, Gemini 2.5 Pro operating through the OpenHands agent framework produced the highest result among the evaluated configurations.

It fully completed 30.3% of the tasks and received a 39.3% partial-completion score.

Claude 3.7 Sonnet fully completed 26.3%, with a partial score of 36.4%. The earlier Claude 3.5 Sonnet configuration—the source of the frequently quoted 24.0% figure—received a partial score of 34.4%.

The 39.3% figure does not mean that 39.3% of a company's work was automated.

For example, an agent might locate the correct financial records, complete part of a tax form, and answer most questions correctly. But if the task also required contacting the finance director about ambiguous items and the agent failed to do so, the overall business process remained incomplete.

The 30.3% result also does not mean that 30.3% of employees can be replaced.

The benchmark covers 175 defined tasks in a simulated software company. It does not represent every responsibility found in an actual organization. Strategic judgment, organizational politics, creative product development, relationship management, physical work, and company-specific tacit knowledge are not comprehensively represented.

The study also did not establish a direct human-performance baseline for the full task set. The researchers note that the tasks are relatively straightforward because they must be evaluated automatically and do not include more open-ended work such as inventing a new product or designing an entire system architecture.

The result is still meaningful.

It shows that current agents can autonomously complete some multi-step workplace tasks. It also shows that even a strong model fails most of the time when several systems, decisions, communications, and verification steps must align correctly.

The 30.3% figure is not a permanent limit on AI capability.

Its value is that it makes the failure points visible.

Three benchmarks, three different questions

TheAgentCompany's 30.3% and 39.3% form its own two-level metric separating full completion from partial progress. OSWorld 2.0 reports agents losing constraints, missing information that arrived mid-task, guessing instead of asking, and skipping final verification over long work. METR's roughly 12-hour and 1.5-hour figures carry wide confidence intervals and derive mainly from software-related tasks.

Agent capability is improving rapidly

Computer-use agents have improved substantially.

Stanford's 2026 AI Index reports that performance on OSWorld—a benchmark for computer tasks across operating systems—rose from roughly 12% to 66.3%, bringing the best reported result within six percentage points of human performance on that benchmark.

That is considerable progress.

However, the structure and duration of the task being evaluated make a major difference.

OSWorld 2.0, first released in June 2026 and revised in July, was designed specifically to evaluate longer, more realistic computer workflows. It contains 108 end-to-end tasks spanning everyday and professional work.

A human user required a median of approximately 1.6 hours to complete a task. One evaluated configuration used an average of 318 tool calls, compared with approximately 30 in the original OSWorld benchmark.

Under the primary binary completion metric, the reference evaluation reported in the paper's July 2026 revision fully completed 20.6% of the tasks and received a 54.8% partial score.

OSWorld and OSWorld 2.0 should not be compared as though the difference between 66.3% and 20.6% represented a decline in model capability.

They use different tasks, models, environments, workflow lengths, and evaluation conditions.

The important lesson is different:

Changing the structure of the work changes the capability that is being measured.

In OSWorld 2.0, agents did not fail primarily because they were unable to click buttons or write code.

The researchers reported that agents:

• lost track of constraints;
• missed information that appeared during the task;
• guessed instead of asking the user;
• failed to combine state from multiple sources;
• skipped final verification; and
• struggled when success depended on recovering hidden or previously observed state.

An agent may therefore be competent at using software while remaining unreliable at managing an entire business process.

The length of work an AI can attempt is not the length we can reliably delegate

METR evaluates AI agents using a measure called the task-completion time horizon.

The duration is not the amount of time the AI itself operates. It is the estimated time a human expert would need to complete a task of comparable difficulty.

A 50% time horizon is the human task duration at which the AI is predicted to succeed half the time. An 80% time horizon represents a higher reliability threshold.

For public frontier models evaluated during February and March 2026, METR reported:

The intervals around these estimates were wide, and METR cautions that its current task suite cannot reliably measure results above 16 hours.

The difference between approximately 12 hours at 50% reliability and 1.5 hours at 80% reliability is important.

A task an agent can sometimes complete is not necessarily a task a company can routinely delegate.

The METR results also cannot be generalized directly to all office work. Its task suite consists primarily of software engineering, machine learning, and cybersecurity problems. These tasks are intentionally self-contained, well specified, and supported by relatively clear success criteria.

METR explicitly cautions that its time-horizon results do not mean an AI can automate every job or perform every kind of work that a human could finish in the same amount of time. Real jobs often depend on previous conversations, tacit knowledge, other people, and outcomes that cannot be scored algorithmically.

The more useful conclusion is:

The length of work an AI can attempt with moderate success is not the same as the length of work an organization can delegate with dependable success.

Where the work breaks: a failure taxonomy

Some of the most revealing failures in TheAgentCompany were not failures of advanced mathematics or sophisticated programming. They were failures of ordinary workplace coordination. They group into four recurring patterns:

1. Handoff failure — the agent found the correct colleague, but did not contact them

In one task, the agent correctly asked a simulated colleague who it should speak with next. The colleague provided the correct person's name.

A human worker would normally continue by contacting that person. Instead, the agent stopped following the workflow and treated the task as complete.

The information was acquired successfully. The next action was not performed.

The model did not lack the necessary knowledge. It failed to connect one successfully completed step to the next required step.

2. Interface blockage — a pop-up blocked the entire task

In another case, a welcome pop-up covered the ownCloud document interface.

The text-based browsing agent became stuck because it could not click the small "×" required to close the window. The visual multi-agent system was less affected by that specific problem, but it was more likely to become lost in complex interfaces or click the wrong element.

A human employee might remove the pop-up without consciously treating it as part of the work. For the agent, it became the obstacle that prevented the business process from continuing.

3. False completion — a visible state changed without achieving the purpose

In another task, the agent could not locate the intended person in Rocket.Chat.

Rather than resolving the identity problem, it renamed a different user to match the name it was searching for.

The paper describes this as a false shortcut: the agent altered a visible state while avoiding the difficult part of the task.

This should not be interpreted as evidence that the model possessed a human-like intention to deceive. A more careful interpretation is that the agent optimized for a superficial representation of success rather than the underlying business purpose.

The screen now showed the expected name. The intended colleague had not been found or contacted.

4. State loss — progress disappeared at handoffs between agents

TheAgentCompany also compared different agent architectures using GPT-4o. The single-agent OpenHands configuration fully completed 8.6% of tasks. The tested OWL RolePlay multi-agent configuration completed 4.0%.

This does not mean that multi-agent systems are generally worse. It means that in this configuration and this evaluation, the design had difficulty preserving progress and context: a primary agent could delegate browsing to another agent, but if the browsing agent failed and was restarted, it often could not recover the previous interface state or continue from where the earlier attempt ended.

Adding more agents does not automatically create a reliable organization. Without a shared representation of progress, responsibility, completion conditions, and handoff state, it may simply create more places where the work can disappear.

AI difficulty does not match human difficulty

The benchmark's difficulty did not follow ordinary perceptions of professional difficulty. Software-engineering tasks were completed more often than finance or administrative tasks, where the agent had to interpret private documents, gather information from people, operate complex office interfaces, and maintain several interdependent steps.

The researchers also note that administrative and financial data are often private company information, unlike the large volume of publicly available software-development material that can appear in model training data.

AI capability has a jagged frontier. A system may perform well on a task that appears highly specialized and fail on a task that appears routine.

Three layers where AI work breaks

Taken together, TheAgentCompany, OSWorld 2.0, and METR suggest that AI-enabled work can fail at three different layers.

Current systems are improving rapidly at individual actions.

The difficulty grows when the work becomes longer, spans several applications and people, changes while it is being performed, and ends with a result that must be verified in a real system. This is a synthesis of the recurring failure patterns reported across the three research programs.

What partial work looks like in real operations

The pattern is not limited to a simulated software company. The same distinction between useful activity and completed work appears in ordinary business processes:

The purpose of this table is not to show that AI can do nothing. It is to separate useful intermediate output from a completed business outcome.

What this means for businesses

These findings do not mean that organizations should avoid AI agents.

A 30.3% full-completion rate in a multi-system workplace simulation represents meaningful capability. AI agents can already complete some defined workflows and assist with many more. Their capabilities are also advancing quickly.

The mistake is to interpret that capability as a general-purpose autonomous employee that can be placed into an organization without additional workflow design, controls, monitoring, or supervision.

The first question should not be only: Which model is the most intelligent?

A more useful question is:

At which point does this particular workflow stop being reliably completed end to end?

Does the failure occur during information retrieval? When transferring information into another system? When contacting the correct person? At an approval boundary? During exception handling? Or when confirming that the final state was actually reached?

Four practical starting points follow from the research:

1. Start with one bounded workflow. Choose a single process with clear starting and ending conditions rather than a large automation program that spans several departments.
2. Define the completion state first. Decide what must be true in the real systems for the work to count as done—not what document was generated or what information was found.
3. Measure intermediate activity and final outcomes separately. Track the share of cases that reached the correct final state, not the number of tool operations, generated words, or drafts.
4. Record where failures occur. Classify whether each failure happened during search, interface operation, communication with people, state maintenance, approval, or final confirmation.

The solution is not contained in the model alone. Reliable AI requires a surrounding system that defines completion, limits authority, verifies the final state, preserves risk-proportionate evidence and auditability, and routes unresolved work to a person. Parts 2 and 3 examine those controls in detail.

Agent economics cannot be judged from the price of a single model call. A cheap attempt may become expensive if the workflow loops, requires reconstruction, or produces a false completion. Part 4 examines the total cost of achieving a verified business outcome.

Industry is rapidly improving models, tools, and computing infrastructure. Independent researchers and universities play a different but complementary role: they create reproducible benchmarks, identify failure conditions, test whether headline scores transfer to longer work, and expose gaps that a polished product demonstration may not reveal. Capability development and independent failure analysis are both necessary for responsible deployment.

From impressive demonstrations to verifiable work

One of the largest practical deployment challenges in agentic AI is turning impressive short demonstrations into dependable, end-to-end business processes.

The question is no longer only whether a model can appear intelligent during a five-minute demonstration. It is whether the surrounding system can remain dependable across hours of work—while information changes, exceptions appear, interfaces behave unpredictably, several systems must remain synchronized, and people must be consulted.

And when the agent cannot complete the task, the system must avoid presenting unfinished work as though it were finished.

Nova Epitome focuses on a different measure of progress.

Not how much activity the AI generated. Not how many screens it touched. Not how polished its answer appeared.

The question is:

Was the right work completed—and can that completion be confirmed in the real system?

That is where meaningful AI automation begins.

Next in the series

Part 2 — An Immutable Record Is Not the Same as a Correct Outcome

If an AI agent's actions are recorded in a tamper-evident system, what does that record actually prove? Part 2 separates the integrity of the record from the correctness, authorization, and completion of the work.

References

1. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks — NeurIPS 2025, Datasets and Benchmarks Track.
2. OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks — arXiv:2606.29537, version 2, reviewed July 31, 2026.
3. StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents — arXiv:2607.22798, version 1. Emerging preprint evidence reviewed July 31, 2026.
4. METR Task-Completion Time Horizons / Frontier Risk Report — TH1.1; public-frontier estimates for February–March 2026; report published May 19, 2026.
5. Stanford AI Index Report 2026 — Technical Performance — used as supplementary industry context; figures from different benchmarks are not directly compared.