AI Evals: A Beginner’s Guide to Measuring What Your System Can Actually Do

An AI demonstration asks whether a system can succeed once. An evaluation asks how often it succeeds, under which conditions, and how failure appears. That distinction matters because generative systems do not behave like conventional functions. The same model may answer a familiar question precisely, mishandle a slight variation, or produce an eloquent response unsupported by the available evidence.

AI evaluations, usually shortened to evals, turn these variable outcomes into product evidence. They help teams compare models, detect regressions, choose architectures, and decide whether a capability is ready for users. The aim is not to manufacture a single authoritative score. It is to build a disciplined method for seeing what the system can actually do.

The Essential Vocabulary

An eval is a repeatable procedure that presents tasks to an AI system and assesses its outputs. The task may be answering a question, extracting fields, selecting a tool, writing code, or completing a multi-step workflow.

• Test case: One input, plus relevant context and an expectation about acceptable behavior.
• Dataset: A collection of test cases intended to represent a capability or risk.
• Ground truth: A known correct answer. It works well for facts, labels, calculations, and extraction, but less well when several responses could be valid.
• Rubric: Explicit criteria used to judge open-ended outputs, such as factual support, completeness, tone, or policy compliance.
• Grader: The mechanism that applies the scoring rule. It may be code, a human reviewer, or another model.
• Metric: A summary of results, such as exact-match rate, task completion, latency, or cost per successful outcome.
• Regression: A loss of previously observed capability after changing a prompt, model, tool, retrieval system, or surrounding code.
• Slice: A meaningful subset of cases, such as long documents, ambiguous requests, or customers using specialist terminology.

One further distinction is foundational. Offline evals run against stored test cases before release. Online evals examine behavior in production through sampled reviews, user outcomes, or controlled experiments. Offline testing is safer and repeatable; online evidence reveals conditions your test set failed to imagine.

The Mental Model: Capability, Conditions, Evidence

A useful eval begins with a capability stated as an observable outcome. “The assistant is helpful” is too vague. “Given an approved policy document, the assistant answers an employee’s question using only supported claims and identifies the relevant policy section” can be tested.

Next, define the conditions. What documents are available? May the system ask a clarifying question? Which tools can it use? What should happen when evidence is absent? A model is only one component of an AI product; retrieval, prompts, tool definitions, permissions, and interface design all shape the result. Evaluate the system users encounter, not merely the underlying model.

Finally, choose evidence proportionate to the decision. If selecting between two extraction models, structured accuracy and processing cost may suffice. If allowing an agent to issue refunds, the eval must examine authorization, amount limits, duplicate actions, tool errors, and safe escalation. Higher-consequence actions demand stronger evidence and narrower tolerances.

What an Evaluation Actually Contains

Suppose you are building an assistant that answers questions from an employee handbook. A useful case contains the employee’s question, the handbook passages available to the system, and criteria for acceptance.

For the question “Can I carry unused leave into next year?”, the expected answer may not be one fixed sentence. The rubric could require the response to state the carryover rule, mention any relevant limit, cite the correct section, and avoid inventing exceptions. If the handbook varies by region, the system should ask which region applies rather than guessing.

Cases should include more than straightforward successes. Add paraphrases, incomplete questions, conflicting passages, obsolete documents, irrelevant retrieved text, and questions whose answers are absent. These cases reveal whether the system possesses a capability or merely recognizes a familiar pattern.

Each case should also preserve diagnostic information: the model version, prompt version, retrieved passages, tool calls, output, score, and grader rationale. Without this trace, a failed score tells you that something broke but not where to look.

Choosing the Right Grader

No grading method is universally best. Match the grader to the shape of the output.

Deterministic checks

Use code when correctness can be expressed mechanically: valid JSON, required fields, numerical tolerances, citation identifiers, tool-call arguments, or successful execution. Deterministic graders are repeatable and inexpensive, but they miss semantic quality. A response can contain every required phrase and still be misleading.

Human review

Humans remain valuable where judgment is genuinely contextual: whether advice is clear, whether a summary preserves emphasis, or whether an interaction feels manipulative. Reviewers need a concrete rubric and examples of borderline cases. Otherwise, disagreement reflects an underspecified standard rather than model quality.

Model-based grading

A language model can compare an output with a rubric at scale. This is useful for supportedness, relevance, and completeness, but the grader is itself fallible. It may favor polished language, overlook subtle contradictions, or vary with prompt wording. Validate it against a sample judged by humans, require a rationale, and use deterministic checks alongside it where possible.

The strongest evaluations often combine methods. Code verifies the response schema and citations; a model assesses whether claims are supported; humans inspect uncertain cases and periodically audit the grader.

Build Your First Eval in Five Steps

1. Name one decision. Decide whether the eval will select a model, approve a release, or diagnose a known failure. Avoid building a general-purpose benchmark without a decision attached.
2. Write the capability contract. State what the system should do, the information it may use, and when it should refuse, abstain, or escalate.
3. Collect representative cases. Begin with real or realistic tasks. Include common requests, important edge cases, and known failures. Remove sensitive information or handle it under appropriate controls.
4. Define scoring before running comparisons. Establish acceptance criteria without seeing which candidate system produced which answer. This reduces the temptation to redesign the rubric around a preferred result.
5. Run, inspect, and classify failures. Do not stop at an aggregate score. Label failures by cause: retrieval missed the source, instructions were ambiguous, reasoning failed, the tool returned an error, or the grader was wrong.

Consider two handbook assistants. System A answers more questions, but guesses when evidence is missing. System B answers slightly fewer and explicitly abstains without support. A single “helpfulness” score might favor A. Separate metrics for answer completeness and unsupported claims reveal the real trade-off. The appropriate choice depends on whether confident fabrication is more damaging than an escalation.

Read Results as a Map, Not a Trophy

An aggregate metric compresses useful variation. Always inspect slices. A system may perform well overall while failing on long inputs, tables, multilingual requests, or tool errors. Those failures can matter more than the average if they affect valuable or vulnerable users.

Also distinguish quality from operating characteristics. A larger model may produce better answers while increasing latency and cost. A retrieval change may improve factual support but omit useful background. Record these dimensions separately, then choose according to the product’s constraints.

When comparing versions, keep the test set, graders, and runtime settings stable. If several elements change together, the result becomes difficult to attribute. Introduce one meaningful change at a time when practical, then examine both improvements and newly created failures.

What to Ignore for Now

Do not begin with public leaderboards. They can illuminate broad model tendencies, but they rarely reproduce your documents, users, tools, permissions, or failure costs. Product-specific evidence should govern product-specific decisions.

Ignore the ambition to cover every conceivable request. Early evals gain value from a small, legible set of consequential cases. Expand the set whenever production reveals a new failure, and retain that case as a regression test.

Do not pursue one universal “AI quality” number. Correctness, supportedness, safety, latency, cost, and user effort are different properties. Combining them prematurely conceals trade-offs.

Finally, resist treating the grader as an oracle. Audit surprising results, inspect rationales, and revise ambiguous rubrics. The eval itself is a product artifact: versioned, tested, and improved as your understanding deepens.

The First Durable Practice

Create a failure library. Whenever a prototype or deployed system behaves badly, preserve the input, context, trace, output, expected behavior, and failure category. Convert the incident into a repeatable test case. Over time, this library becomes more valuable than a generic benchmark because it records the precise ways your system encounters reality.

The deeper opportunity is not merely better testing. Evals make capability development cumulative. Prompts stop being judged by memorable demonstrations. Model upgrades stop being acts of faith. Architecture decisions become comparisons against explicit requirements. The team acquires a shared language for what “better” means—and a mechanism for noticing when progress in one dimension quietly creates risk in another.

This post was drafted with AI assistance and reviewed against our editorial policy before publication. Corrections are made at the source, on the page, with the date shown.