AI engineering · Field guide
Choose Agent Work by How Clearly You Can Verify It
Coding agents are moving from suggestions to assigned work. The useful question for an engineering lead is not how much work an agent can attempt, but whether the result can be checked against the requirement before it reaches users.
By Nythrex team4 min read
A backlog ticket can look small and still be a poor candidate for autonomous work. “Update the account flow” may span several services, depend on unstated product rules, and have no test that proves the behavior is right. A narrow bug report with a reproducible case may be much safer to hand over, even if its code change is larger.
That distinction matters as delegation grows. GitHub’s October 28, 2026 Octoverse report says more than 40% of pull requests on GitHub started with an agent’s first draft in September, compared with 17% in March. But growing volume does not tell a team which tasks to delegate. Recent benchmark research offers a more useful lens: inspect the structure of the task and the evidence available to verify it.
- Share of GitHub pull requests that started with an agent’s first draft in September 2026, up from 17% in March. GitHub Octoverse, October 2026.
- >40%
- Share of agent-started pull requests changing multiple files, compared with 79% of pull requests without detected agent involvement. GitHub Octoverse, October 2026.
- 87%
- AUC reported for predicting task difficulty from static task features in the CoderForge-Preview study. Al-Haque and Johnson, 2026; this is a study result, not a score for any particular team or agent.
- 0.863
A task’s shape is a useful warning signal
In an October 2026 ESEM paper, researchers Ebtesam Al-Haque and Brittany Johnson studied coding-agent trajectories across 51,000 software issue tasks. They report that structural features, especially patch fragmentation and repository scale, predict task difficulty in their dataset. The finding is not a universal screening rule: it measures associations in a benchmark collection, not guaranteed success on a team’s private backlog.
A separate October 2026 benchmark, AgentSWE, tests agents building, editing, and optimizing other agents. The authors report recurring gaps between development cases and held-out cases, plus outputs that looked complete but failed when executed. This is a specialized benchmark, so it does not establish failure rates for ordinary feature work. It does reinforce a practical point: a clean-looking patch or a confident completion message is weak evidence unless a check exercises the requirement.
Choose delegated work by the quality of the evidence you can inspect, not by how small the ticket sounds.
The verification path belongs in the estimate
When a task crosses modules, depends on implicit rules, or has only a handful of easy examples, review gets harder. The agent may produce more code across more files, as GitHub’s figures suggest, while the engineer still has to reconstruct intent and design checks. That is a work-allocation problem: the apparent coding time saved may be consumed by clarification, test design, and tracing behavior across the repository.
This suggests a delegation boundary. Start with work whose expected behavior can be stated, exercised, and reviewed without relying on undocumented judgment: a reproducible defect, a bounded migration with compatibility checks, or tests for an agreed behavior. Keep tasks involving ambiguous product decisions, broad architectural trade-offs, or high-consequence changes under close human direction until the team can specify and verify them. These are starting categories, not guarantees.
Use a verification-first delegation check
Before assigning an agent a task this month, ask the engineer who owns it to write down four things: the behavior to change, the cases that must continue to work, the repository areas likely involved, and the evidence that will count as done. Then classify the task informally. A clear requirement plus strong executable checks is a good pilot. A fragmented change across a large or poorly understood codebase needs a human decomposition step. If nobody can say what would disprove the result, keep the work exploratory and require an engineer to make the decision.
For a small pilot, compare similar tasks with and without agent help. Record the engineer time spent clarifying, preparing context, checking behavior, revising the patch, and maintaining it after merge. Review failures as evidence about the task definition or test boundary, not just the model. Do not count a task as finished because an agent opened a pull request or reported success; count it when the team’s normal acceptance checks pass and the responsible reviewer agrees the change meets the requirement.
Sources
- GitHub Blog — Octoverse: Commits, pushes reach new highs in 2026 as developers build with agents (October 28, 2026)
- Al-Haque and Johnson — What Makes Software Issue Resolution Tasks Difficult for Agents? (ESEM 2026; August 18, 2026)
- Wang et al. — AgentSWE: Can Coding Agents Build the Agent You Actually Want? (October 9, 2026)
Frequently asked questions
How should engineering teams choose tasks for coding agents?
Start with tasks that have a clear expected behavior and checks capable of showing whether the change meets it. Treat broad, fragmented, or poorly specified work as needing human decomposition and closer direction.
Does a passing test suite prove an agent’s change is correct?
It is evidence, but only for the cases the suite covers. The AgentSWE study reports gaps between development and held-out cases in its benchmark, so teams should check requirements and relevant edge cases as well.
What should teams measure in an agent pilot?
Track the full engineer effort around each change: clarification, context preparation, review, revisions, and follow-up maintenance, alongside whether acceptance checks pass. A pull request or agent completion claim alone does not establish that work is done.
Read next
Stop guessing.
Start seeing.
No card and no subscription: paid plans have not launched yet.