workflowVerification6 min read

Evals for the LLM Features You Ship

Our take

A feature that calls a model does not give the same answer twice, so a green unit test says very little about it. The eval set that protects it cannot be written in a planning doc. It gets harvested from what the feature actually did to real users.

A loop for the LLM features in your own product: log traces, read them, turn real failures into permanent eval cases, score a sample of live traffic, and only then add guardrails. The agent under test here is the one your customers talk to, not the one writing your code.

On this page10 parts

01Not the Same as Golden-PR Evals
02The Problem
03The Loop
04Two Sets You Write by Hand
05Grade With Code First
06What to Score Beyond Correctness
07Tooling
08Where It Fits in the Stack
09When Not to Use It
10Source

Not the Same as Golden-PR Evals

The two share vocabulary and measure different things. Run both if you have both problems.

Golden-PR Evals score your coding agent setup.The tasks come from your merged pull requests, the thing under test is your harness, and a failure means your team ships worse code.
This entry scores what your product does.The cases come from your production traces, the thing under test is your prompt, model, retrieval and tools, and a failure lands on a customer.

The Problem

A prompt edit, a model upgrade or a retrieval change can each move behavior without changing a single type or breaking a single unit test. You click through a few cases, it feels right, you ship. Every change is a coin flip you cannot see the result of.

The usual fix is to sit down and write a benchmark. That is better than nothing, and it is also where teams get stuck. A curated set tests the world you imagined during planning, so it stays green while real users send phrasings and edge cases you never thought of. The authors of EvalGen (opens in a new tab) describe why this cannot be solved by thinking harder: you need criteria to grade outputs, and grading outputs is how you discover the criteria.

The Loop

1. Log traces. You cannot grade what you did not record. Capture the input, the retrieved context, the tool calls and the final output of each interaction.

2. Read them. A person who knows the domain reads real traces and writes open-ended notes on what went wrong. Hamel Husain calls this error analysis. Keep going until new traces stop teaching you new failure modes.

3. Name the failure modes. Cluster the notes into a taxonomy and count each category. The counts tell you which evals to write first. Metrics chosen before this step tend to be generic scores that do not move when the product gets better.

4. Promote failures to eval cases. Each graded failure becomes a permanent test, run on every change to a prompt, a model or the retrieval pipeline.

5. Score live traffic. Sample production continuously, which is what online evals means. The offline set is your regression net. The online stream is how you find out what to add to it.

6. Then add the guardrail. A guardrail is an eval that runs in production and is allowed to block, retry or escalate. It comes after the eval that proves you need it, so the same suite can tell you what it catches and what good requests it now blocks.

Two Sets You Write by Hand

Harvesting has two exceptions, and teams often get them backwards by authoring a quality benchmark and skipping safety entirely.

Cold start.Before launch there are no traces, so write cases by hand, often with synthetic data. Treat them as scaffolding and replace them as real failures arrive.
Safety.You cannot wait for a real breach to log it. Invented policies, leaked PII and injected instructions get written up front and adversarially, against something like the OWASP Top 10 for LLM Applications (opens in a new tab).

Grade With Code First

Most failures that hurt are checkable without a model: did it cite a document that exists, does the JSON parse, did it call an allowed tool, is that order number real. Code is cheap and has no opinions of its own.

TypeScript

1// Harvested from a production trace: the assistant stated a refund window
2// that does not exist. Check the citation, not the wording, because a
3// correct answer may also mention the same number of days.
4test('every stated policy traces to a real document', async () => {
5  const result = await runAssistant('Can I still get a refund after 90 days?');
6
7  for (const claim of result.policyClaims) {
8    expect(claim.citedDocId).toBeTruthy();
9    expect(await docExists(claim.citedDocId)).toBe(true);
10    expect(claim.foundInDoc).toBe(true);
11  }
12});

Save the LLM judge for what code cannot decide, such as tone or helpfulness, and treat it as a system that needs its own checking. Published work documents position bias (opens in a new tab), where swapping the order of two answers changes the verdict, and self-preference (opens in a new tab), where a model favors its own outputs. Give the judge the source document and an expected answer to anchor against, pin its model, and measure how often it agrees with your human labels.

What to Score Beyond Correctness

Consistency.Run the same input several times. Some variance is fine, and only you can say how much your users will notice.
Latency and cost.Record both per run, so an upgrade that scores better but costs more is a visible tradeoff instead of a surprise on the invoice.
The experience around the answer.For a Frontend team this is the part nobody else will check. A correct answer that streams into a broken layout or hides a failed tool call is still a failure, so drive the real UI with the Browser Verification Loop as well as scoring the text.

Tooling

The tool matters less than the loop. All of these can run evals against a dataset, so choose on where your traces live and how you want to host them.

Langfuse (opens in a new tab).Tracing, datasets, online evaluators, annotation queues and a CI gate in one place. The core is MIT licensed and self-hostable. ClickHouse acquired it in January 2026 and both companies said licensing would not change.
Braintrust (opens in a new tab).A hosted platform with experiments, scorers, online scoring of production traces and a GitHub Action that posts results on the pull request.
Arize Phoenix (opens in a new tab).OpenTelemetry-based tracing plus evals, self-hosted. Check the license if it matters to you: Elastic License 2.0, which is not an OSI-approved open source license.
Promptfoo (opens in a new tab).Declarative configs run from the CLI and in CI, with evals running locally, and red teaming built in, which makes it a natural home for the safety set. Now part of OpenAI, and still MIT licensed.
Inspect (opens in a new tab).An open source Python framework from the UK AI Security Institute, co-developed with Meridian Labs. Heavier than the others and built for rigorous model evaluation. Reach for it when the stakes justify the extra weight.

Where It Fits in the Stack

Pairs with Building Effective Agents.Each pattern has its own failure modes, and evaluator-optimizer needs a reliable evaluator before it is worth anything. This loop is how you get one.
Borrows its grading discipline from Golden-PR Evals.Deterministic checks first, a calibrated judge second, humans reading transcripts on a schedule. The rules carry over even though the thing under test does not.
Belongs in the Frontend AI conversation.It is about what your product does, not how your org ships.

When Not to Use It

A low-stakes internal tool.If a wrong answer costs a shrug and a retry, reading some traces now and then is enough. The rule of thumb: if a regression would generate a support ticket, it needs evals.
Before you have logging.A dashboard of scores with no traces behind it tells you something moved and gives you no way to find out why. Start with step one.
As one aggregate number.A single percentage invites optimizing the percentage. Report by failure mode.