Golden-PR Evals

The Problem

Public benchmarks tell you nothing about your repo. A 70% score on SWE-bench Verified and a 30% pass rate on your codebase are both real numbers about the same agent. Your repo has internal libraries that are not on npm, conventions no benchmark encodes, and a review bar that would reject half the diffs the public set accepts.

So teams fly blind. Someone rewrites CLAUDE.md and it "feels better". Someone adds four skills and nobody knows whether any of them fired. A new model ships and the team upgrades on release-day vibes. When the agent produces a bad week, there is no way to tell whether the cause was the model, the context file, an MCP server that stopped loading, or the tasks simply being harder that week.

The stakes are not theoretical. CodeRabbit reviewed 470 open-source pull requests and found AI-co-authored PRs carried roughly 1.7x more issues than human-only ones, 10.83 per PR against 6.45, with logic and correctness problems up 75% and performance issues far higher still. "An agent wrote it" is not a quality claim. It is a reason to measure.

And measurement is where the leverage is. LangChain moved a deep-agents harness from 52.8 to 66.5 on Terminal-Bench 2.0 with harness changes alone, no model change. That 13.7-point jump is the size of a model generation, and it is only reachable if you can see which change moved the number.

What a Task Actually Is

Anthropic's evals post gives the vocabulary, and it is worth using the same words as everyone else:

• Task.One scenario: a starting repo state, a prompt, and success criteria.
• Trial.One run of the agent on one task. Agents are stochastic, so one trial is an anecdote.
• Grader.The thing that decides whether the trial passed. Code, model, or human.
• Transcript.The full record of what the agent did, which is where the actual diagnosis lives.
• Outcome.The end state of the repo and the environment, which is what you should grade whenever you can.

Build the Suite From Your Own History

Your merged PRs are a labeled dataset nobody had to write. For each task, check out the parent commit of a merged PR, hand the agent the issue text or the PR description as the prompt, and grade against what the merge actually produced.

Practical rules that keep it honest:

• Shallow clone at the parent commit.A full clone carries the merge commit and its tests somewhere in the object graph, and an agent that finds the answer scores well for the wrong reason.
• Grade outcomes, not diff similarity.There are many correct implementations. "Do the tests from the merged PR pass" is a real criterion. "Does the diff look like the merged diff" is not.
• Twenty tasks is enough to start.Roughly 40% routine work the agent should never fail, 40% the kind of change that actually goes wrong in your repo, 20% known-hard cases you expect to fail today. A suite with no failures measures nothing.
• Add a task every time an agent burns an afternoon.The failure you just debugged by hand is the highest-value task in the suite.

Grade It in Three Layers

One judgement channel is not enough, and a single 1-to-10 score from a model is the worst of the three.

• Deterministic first, always.Does it build, do the tests pass, does the type-check survive, is the bundle under budget, does the a11y check return clean, did it touch files it had no business touching. Anything with an exit code goes here.
• A model judge for what has no exit code.Convention adherence, whether the diff is minimal, whether the final change still matches the plan the agent wrote at the start. Give the judge a rubric with named dimensions and a per-dimension verdict, not a vibe score. Calibrate it against a set you labeled by hand and track agreement; strong judges land above 80% agreement with human preference, and a judge you never calibrated is just a second opinion with no accountability.
• Human spot-checks on a schedule.Read ten transcripts a month. The aggregate number tells you something moved. Only the transcript tells you why.

Run It More Than Once

A single trial per task produces a number that changes when nothing changed. Run k trials and report both metrics, because they answer different questions.

• pass@kis "did at least one of k attempts succeed". It measures capability, and it rises with k.
• pass^kis "did all k attempts succeed". It measures reliability, and it falls with k.

The gap is brutal and it is the entire argument for measuring reliability separately. An agent that succeeds 75% of the time on a single trial passes all three attempts only about 42% of the time. If a human reviews every diff, pass@k is the number you care about. If the run is unattended, pass^k is the only one that means anything, and it is the number that decides which rung of the autonomy ladder that workflow has earned.

The Frontend Case Is the Hard One

Backend tasks have exit codes. A frontend change can compile clean, pass every unit test, and still ship a broken 768px layout, a modal that traps focus, or a component built out of raw hex values instead of your tokens. Execution success is not the criterion your users experience.

So the grader has to open the page. This is the Browser Verification Loop pointed at a scoring function instead of at a single change:

• Deterministic where you can force it.Accessibility-tree snapshots, computed styles compared against design tokens, console error count, a Playwright assertion on the interaction, an LCP budget under throttling. These are cheap, stable, and they carry no model judgement.
• A rubric judge for visual fidelity.Screenshot at each breakpoint, judged component by component against named dimensions: spacing scale, type weight, token compliance, state coverage, layout at 360, 768 and 1200. Research systems converging on this pattern grade rendered appearance and interactive behavior as separate scores, and so should you.
• Interaction, not just pixels.A page can look right and do nothing. Drive the flow, then assert the resulting state.

Wire It Into CI

claude -p is the whole harness. One subtlety decides whether the suite measures anything useful:

1# Wrong for an eval: --bare skips hooks, skills, plugins, MCP servers and
2# CLAUDE.md, which is exactly the setup you are trying to measure.
3claude --bare -p "$TASK" --allowedTools "Read,Edit,Bash"
4
5# Right: load a pinned configuration explicitly, so the run is reproducible
6# and the thing under test is the thing that loaded.
7claude -p "$TASK" \
8  --settings ./evals/settings.json \
9  --plugin-dir ./evals/plugins/frontend-factory \
10  --mcp-config ./evals/mcp.json \
11  --permission-mode dontAsk \
12  --output-format json > "trial-$TASK_ID-$TRIAL.json"

--permission-mode dontAsk denies anything outside your allow rules and the read-only command set, which is the right posture for a run nobody is watching. The JSON payload carries result, session_id and total_cost_usd, so cost per task falls out of the same file you grade. Add --json-schema when you want the agent's own report in a fixed shape rather than prose.

Then gate on the delta, not the absolute:

• Run on merge to main and on every change to the harness itself.CLAUDE.md, skills, hooks, plugin version, MCP config, model. Not on every PR, because a 20-task suite at 3 trials is real money and real minutes.
• Fail on regression against the stored baseline.A drop in pass^k on the routine tier is a broken harness until proven otherwise.
• Check plugin_errors and mcp_server_errors in the system/init event.A plugin that silently failed to load turns your eval into a measurement of the default setup, and the score will move for a reason you will never find by reading transcripts.
• Store transcripts as artifacts.The score tells you there is a problem. The transcript is the only thing that tells you what it was.

Why This Matters for Senior Teams

• It settles arguments that otherwise run on seniority."Opus or Sonnet for this repo", "is the new skill helping", "did the CLAUDE.md rewrite work". These become experiments with a result instead of debates with a winner.
• Harness work becomes fundable.A 13.7-point swing from harness changes alone is worth a sprint. Nobody approves that sprint without a before number.
• It closes the delegation gap honestly.Anthropic's 2026 trends report found engineers using AI in roughly 60% of their work while fully delegating only 0 to 20% of tasks. That gap is a trust problem, and trust in an unattended run needs a reliability figure, not a good week.
• It is the security answer's twin.The autonomy ladder decides what the agent is allowed to do. The eval suite decides what it has earned. Raising a workflow's rung because pass^k held at 3 trials across 20 tasks is a defensible decision. Raising it because the last few sessions went well is not.
• It survives the people who built it.A suite in the repo outlives the engineer whose intuition was carrying the team's agent setup.
• It is what an acceleration claim rests on.Faster delivery only counts if quality held, and quality holding is a measurement, not an assertion.

Where It Fits in the Stack

• Sits above the Browser Verification Loop.The loop verifies one change. The suite verifies that the loop, the context file and the skills are still doing their job across many changes.
• Scores CLAUDE.md and AGENTS.md.Run the suite with and without the context file. Whatever the difference is, that is what the file is worth.
• Gives skills and plugins a version number that means something.Bump the plugin, run the suite, record the score against the tag. Rolling back becomes a decision with evidence behind it.
• Decides autonomy rungs.Reliability data is the input to how far a given workflow is allowed to run unattended.
• Feeds the Writer / Reviewer pattern.The rubric your judge uses and the checklist your reviewer session follows should be the same document.
• Belongs in the Factory conversation.For teams shipping frontend at pace, this is the layer that makes "we accelerated delivery" a sentence with a number behind it.

When Not to Use It

• Not before you have a harness worth measuring.Write the CLAUDE.md, add the verification loop, then measure. An eval suite over an unconfigured setup measures the model and nothing else.
• Not on every pull request.The suite is a regression gate on the harness, not a pre-merge check on application code. Your normal tests already own that.
• Not as a single aggregate number.A suite that reports one percentage invites optimizing the percentage. Report per-tier and per-dimension, and keep the hard tier failing until it genuinely should not.
• Not with an uncalibrated judge.A model judge that was never checked against human labels will drift, and it will drift in the direction of whatever the agent happens to produce.
• Not against anything real.Tasks run with write access and no supervision. Shallow clones, throwaway containers, no production credentials in the environment.

Source

• Demystifying evals for AI agents, Anthropicanthropic.com (opens in a new tab)
• Run Claude Code programmaticallycode.claude.com (opens in a new tab)
• Claude Code in GitHub Actionscode.claude.com (opens in a new tab)
• 2026 Agentic Coding Trends Report, Anthropicresources.anthropic.com (opens in a new tab)
• State of AI vs Human Code Generation, CodeRabbitcoderabbit.ai (opens in a new tab)
• Improving deep agents with harness engineering, LangChainlangchain.com (opens in a new tab)