Engineering · Testing
Coding agents are changing who writes the tests. Can you trust the result?
Coding agents can produce tests alongside application changes, but a larger test diff does not automatically mean stronger protection. Recent research offers a more useful question for engineering leaders: what testing work is changing, and what evidence would show that the tests help?
By Nythrex team3 min read
When an agent opens a pull request with new tests, it is tempting to count the tests as extra assurance. But test code can be present and still miss the behavior that matters, assert the wrong thing, or make maintenance harder. The difference between activity and useful coverage matters more as teams delegate more implementation work.
- Merged pull requests analyzed across 391 Java repositories, January 2025–April 2026. Milanese et al., arXiv, 2026.
- 114,134
- Within repositories, the odds that an agent-supported pull request changed test code were 1.65 times those for human-authored pull requests. Milanese et al., arXiv, 2026.
- 1.65×
1. A pooled average can hide how teams work
Roberto Milanese and co-authors examined 114,134 merged pull requests in 391 Java repositories. The pooled comparison found that agent-supported and human-authored pull requests changed test code almost equally often. But when the researchers compared the groups within repositories, controlling for change size, agent-supported pull requests had 1.65 times the odds of changing test code. They also added tests more often and modified or deleted existing tests less often (Milanese et al., 2026).
That difference is a reminder to ask what a metric compares. Repositories have different testing cultures and different rates of agent adoption. A pooled result can blur those differences; a within-repository comparison is more informative, though still observational. The study identifies associations in its sample, not a guarantee that assigning an agent causes better testing.
2. More test changes are not the same as better tests
The paper found no difference in most structural characteristics of the tests agents added, and no difference in production-change coverage among the 95 repositories the researchers could build. In test-only pull requests, however, agent-authored tests covered less of the code they exercised. The authors also describe test-quality evidence as uncertain: differences in test smells were borderline and did not reproduce at the individual test-method level (Milanese et al., 2026).
Coverage is a useful signal about which lines executed, not whether an assertion would catch a real regression. A test can run a line without checking the expected result. Conversely, a focused test can protect a critical contract without maximizing a coverage percentage. Read the test body and the behavior it is meant to preserve.
| Review question | Useful evidence | Common blind spot |
|---|---|---|
| What behavior is protected? | A named requirement, boundary, or defect scenario | A test that merely repeats the implementation |
| Would a regression fail it? | Assertions that check meaningful outputs or side effects | Execution without a meaningful assertion |
| What risk remains? | Relevant untested paths and limits stated plainly | Treating line coverage as correctness |
A test diff is evidence of testing activity; the assertion is evidence of what the test checks.
3. Set a review standard around behavior and scope
The delivery context is changing quickly. GitHub reports that more than 40% of pull requests on its platform began with an agent’s first draft in September 2026, up from 17% in March; its report also notes that agent-started pull requests more often changed multiple files. These are GitHub telemetry figures with detectable agent signals, not a universal measure of software teams. They suggest why a consistent test-review standard matters as more work arrives in agent-assisted form (GitHub, 2026).
What to do on Monday
- 1
Pick a representative change
Choose one recent agent-supported pull request that changes behavior, plus a comparable human-authored change in the same repository. Keep the review focused on what tests changed and why. - 2
Trace each important test
For each new or changed test, write down the behavior it protects, the assertion that would fail on a regression, and the risk it leaves open. Ask the author or agent to explain intent where the diff does not make it clear. - 3
Track signals together
Compare test-change frequency and coverage with escaped defects, flaky-test burden, and reviewer findings over time. Keep repository and change-size context visible, and treat small samples as prompts for investigation rather than verdicts.
Sources
Frequently asked questions
Do coding agents write better tests?
Recent research found that agent-supported pull requests add tests more often within the studied Java repositories, but it did not establish that those tests are generally better. The authors describe test-quality evidence as uncertain.
Is code coverage enough to judge agent-written tests?
No. Coverage indicates which code ran, while review must also check whether assertions verify meaningful behavior and would expose a regression. Use coverage as one signal alongside test intent and observed defects.
Read next
Stop guessing.
Start seeing.
No card and no subscription: paid plans have not launched yet.