Evaluate AI Coding Agents at the Patch, Not the Promise
When an AI coding agent says “done,” you have a conversational state—not yet a mergeable change. A patch is useful only when its behavior, regressions, scope, and evidence can be checked.
The practical answer is to turn each request into a reproducible patch-evaluation task. Specify the contract, add a fail-to-pass test for the new behavior, preserve pass-to-pass regression tests, and inspect the diff for out-of-scope changes.
Four signals for one patch
Start with a precise specification. “Improve search speed” is not an eval task. State the inputs, allowed paths, success criteria, and forbidden side effects so two reviewers can reach the same pass/fail decision.
Next, use a fail-to-pass test: it fails before the change and passes after the correct patch. Pair it with pass-to-pass tests for important existing behavior. OpenAI describes this distinction in its SWE-bench material because solving the new issue does not prove that unrelated behavior survived.
Finally, inspect the diff. Passing tests do not justify an unrelated dependency, a temporary bypass, or a change to files outside the contract. Code graders are fast; diff review is better at exposing intent and blast radius.
A minimal task contract
id: normalize-email-whitespace
request: "Trim login email whitespace before validation"
allowed_paths:
- src/auth/email.ts
- tests/auth/email.test.ts
fail_to_pass:
- trims leading and trailing whitespace before validation
pass_to_pass:
- rejects an empty email
- rejects an invalid email
constraints:
- do not change the database schema
- do not add a dependency
- report the final diff and test command
The point is not to create a large test catalog. It is to avoid hidden criteria that a reasonable agent could not infer. OpenAI’s 2026 audit of coding evaluations highlights overly strict tests, underspecified prompts, and low-coverage tests as task-quality problems. Repeated 0% performance should trigger an audit of the task and grader before a verdict on the model.
Capability and regression serve different purposes
Anthropic distinguishes capability evals from regression evals. Capability evals target work the agent still struggles with; regression evals protect behavior that already works.
A workable loop is:
- Mine real bugs, support tickets, and release checks for 10–20 tasks.
- Give each task an isolated fixture and a human-verified reference solution.
- Separate hidden implementation details from explicit fail-to-pass and pass-to-pass behavior.
- Run multiple trials when consistency matters.
- Read failed transcripts and diffs to separate grader defects from agent defects.
- Graduate solved capability tasks into the regression suite and add harder tasks.
A public benchmark score is not a product-quality proxy. OpenAI’s later analysis of SWE-bench contamination and task validity is a useful reminder that evaluations themselves need testing.
Keep gates separate
An internal gate can stay simple:
task_pass = fail_to_pass passes AND pass_to_pass passes
scope_pass = no edits outside allowed paths
review_pass = no critical security, deletion, or bypass finding
release_gate = task_pass AND scope_pass AND review_pass
Do not let a high test score cancel a binary scope or security failure. OpenAI’s Graders API supports string checks, similarity, and multi-graders, but the right mix depends on failure cost. Use model graders for qualities such as explanation or instruction following; retain code checks and human review for destructive, security-sensitive, or financial changes.
A prompt for Codex or Cursor
Summarize the success criteria and allowed paths before editing.
Read the relevant implementation and tests, then explain what each new test proves.
Run fail-to-pass and existing pass-to-pass tests separately.
After tests pass, inspect the diff for out-of-scope files, dependencies, and bypasses.
If anything fails, classify it as task, environment, or implementation and stop.
Report changed files, commands, test results, and remaining uncertainty.
The important move is not asking the agent to “think harder.” It is fixing the shape of the evidence. Repository instructions can guide behavior, but the task contract and graders decide whether the result passes.
A senior engineer’s trade-off
Patch evals cost time: fixtures need maintenance and transcripts need sampling. Applying the same process to a cosmetic one-line change can slow a team down. Authentication, billing, and migrations deserve isolated environments, regression suites, security checks, and human approval.
Risk-tier the process. For low-risk changes, use focused tests and diff review. For medium-risk work, add regression groups and repeated trials. For high-risk changes, require isolation and a human gate. The goal is not to remove autonomy; it is to make recovery and rejection explicit.
FAQ
Are passing tests enough?
No. Check the new behavior, regressions, and whether the diff stayed within the contract.
How many tasks should we start with?
About ten real tasks that the team already checks manually is enough to begin. Representativeness matters more than a round number.
Does an LLM judge replace humans?
No. Anthropic recommends combining code-based, model-based, and human graders, with transcript review and periodic calibration.
Can we compare public coding benchmarks?
Yes, cautiously. Record model, scaffold, seed, data exposure, and task-quality caveats; internal tasks are more directly useful for shipping decisions.
Closing
The real productivity of an AI coding agent is not how convincing its answer sounds. It is how cheaply the team can decide whether a patch is correct. Pick one task and record its specification, fail-to-pass test, pass-to-pass tests, and diff boundary. That small contract lets future model and prompt changes be judged by evidence instead of intuition.