How to Build a Capability Probe for an AI Model Before Designing the Product

AI products are often designed backward. A team imagines an interface, selects a model, writes an elaborate prompt, and then discovers that the underlying capability is unreliable precisely where the product needs confidence.

A capability probe reverses that sequence. It is a small, disciplined experiment that asks what a model can do under realistic conditions before a team commits to features, workflows, or positioning. The concrete outcome is not a benchmark score. It is a defensible product boundary: which tasks to automate, which to assist, which to escalate, and which to exclude.

Step 1: Define the decision the probe must unlock

Begin with a product decision, not a general question about whether a model is “good.” Useful probes resolve choices such as whether an insurance assistant may draft claim summaries, whether a research tool can extract findings from technical documents, or whether a support system should answer customers directly.

Write the decision as a sentence: We will use this probe to decide whether the system should draft responses from approved support documents for human review. Then name the consequence of being wrong. A fabricated return policy creates a different risk from awkward phrasing.

Specify four elements:

• User: the person performing or receiving the work.
• Artifact: the input the system will actually see.
• Action: the transformation or judgment required.
• Consequence: what happens when the output is wrong.

Common mistake: testing an abstract ability such as summarization. Summarizing a meeting, a medical history, and a contract are materially different jobs. The artifact, required omissions, and cost of error determine whether capability becomes useful.

Step 2: Decompose the workflow into testable task units

A workflow usually conceals several capabilities. “Handle a support ticket” may require identifying intent, finding the governing policy, extracting account facts, choosing an allowed action, drafting a reply, and detecting when escalation is necessary. A single end-to-end score will not reveal which link failed.

Map the workflow, then isolate units that can be evaluated independently. Each unit should have a visible input, a requested operation, and observable acceptance criteria.

Common mistake: decomposing work according to interface screens rather than reasoning requirements. A screen may combine retrieval, calculation, and policy interpretation. Probe those mechanisms separately before recombining them.

Step 3: Assemble a small but adversarial case set

Do not begin with hundreds of examples. Begin with a compact set whose composition you understand. Draw cases from real artifacts when permission and privacy controls allow. Remove personal data, preserve operational complexity, and record why each case matters.

Include several case types:

• Routine: clear inputs representing ordinary work.
• Boundary: cases just inside or outside an allowed rule.
• Incomplete: essential information is absent.
• Conflicting: sources or instructions disagree.
• Distracting: irrelevant but persuasive details appear.
• Exceptional: rare situations requiring escalation.

For example, a return-policy probe should not contain only obvious eligible and ineligible purchases. Include a purchase near the deadline, an item covered by a category exception, an obsolete policy passage, and a message that never states the purchase date. These cases expose whether the model knows when the evidence does not support a decision.

Common mistake: building the set from cases the team knows the model handles well. This produces a demonstration, not a probe. Include examples that could disprove the intended product.

Step 4: Create a scoring contract before running the model

Define success without looking at outputs. Otherwise, evaluators unconsciously reward articulate answers and reinterpret requirements around what the model produced.

For each case, prepare a reference sheet containing required facts, prohibited claims, acceptable variants, escalation conditions, and severity of failure. Avoid demanding one exact phrasing when several answers would be operationally valid.

A useful scoring contract separates dimensions:

1. Factual grounding: Are claims supported by the supplied material?
2. Task correctness: Is the classification, extraction, or action right?
3. Constraint compliance: Were policy and format requirements followed?
4. Uncertainty behavior: Did the system identify missing or conflicting evidence?
5. Usability: Can the intended user act on the output without reconstructing it?

Mark certain errors as disqualifying. In a policy assistant, inventing an exception may be unacceptable even when the remainder of the response is useful. Averages can conceal such failures.

Common mistake: scoring style and correctness together. A polished falsehood can outrank a terse, correct answer. Keep presentation subordinate to operational validity.

Step 5: Run controlled variants, not prompt improvisation

Choose a baseline configuration and record it: model version, system instructions, tools, retrieved context, output format, and sampling settings. Then vary one meaningful element at a time.

Useful comparisons include a direct prompt versus a structured output, no retrieval versus retrieval from approved documents, or a single pass versus a second verification pass. If the model changes alongside the prompt, retrieval method, and tool access, you will not know what caused the improvement.

Consider a research-extraction probe. The baseline asks for the intervention, sample, and reported outcome from a document. A second configuration requires each field to include supporting text. A third permits the model to return not stated. If unsupported completion falls after the third change, the product insight is not merely that the prompt improved. It is that abstention must be a first-class interface state.

Common mistake: repeatedly editing prompts around memorable failures. This overfits instructions to the probe set. Freeze the cases, run named variants, and reserve untouched cases for confirmation.

Step 6: Review blindly and diagnose mechanisms

Hide configuration names during review. If evaluators know which output came from the favored model, expectation can influence judgment. For consequential domains, use at least two perspectives: someone who understands the subject matter and someone who understands the operating workflow.

Do more than label answers correct or incorrect. Assign each failure to a mechanism:

• Relevant evidence was not retrieved.
• Evidence was present but misread.
• A calculation or transformation was incorrect.
• An instruction conflicted with a learned convention.
• Missing information was filled with an assumption.
• The answer was correct but unusable in the next workflow step.

This diagnosis determines the remedy. Retrieval failures may require better document segmentation or metadata. Arithmetic may belong in a deterministic tool. Ambiguous cases may require a human decision. No prompt can repair absent source material.

Common mistake: treating every failure as a model failure. The surrounding system often creates the weakness. Equally, do not disguise reasoning failures with elaborate orchestration before proving that the added complexity changes outcomes.

Step 7: Convert findings into a product boundary

End with a decision matrix, not a leaderboard. Group tasks according to observed reliability and consequence.

• Automate: bounded tasks with observable outputs and recoverable errors.
• Assist: useful drafts where a person can efficiently verify the evidence.
• Escalate: ambiguous, exceptional, or high-consequence cases.
• Exclude: tasks whose failures are difficult to detect or whose inputs are insufficient.

Suppose the support probe shows strong policy retrieval and drafting, but weak handling of exceptions. The right product is not an autonomous agent with a broader prompt. It is a drafting system that cites the governing passage, asks for missing facts, and routes exception categories to a specialist.

Write the resulting boundary as an operational rule: The system may draft responses only when it identifies a current policy passage and all required account facts; otherwise it must request information or escalate. This statement can guide interface design, logging, access controls, and launch criteria.

Common mistake: turning a promising result into a universal capability claim. A probe supports conclusions only within its artifacts, task definitions, and tested conditions.

Step 8: Preserve the probe as a living product instrument

Store cases, reference sheets, configurations, outputs, reviewer decisions, and failure labels together. When the model, prompt, retrieval index, policy corpus, or toolchain changes, rerun the probe. Add representative production failures, but keep part of the case set untouched so improvement is not merely adaptation to known examples.

The deeper value of a capability probe is architectural clarity. It reveals where probabilistic judgment creates leverage, where deterministic software should take control, and where a person remains essential. Designed early, it prevents the interface from promising more than the system can reliably deliver—and exposes smaller, sharper opportunities that a feature-first process would never see.

This post was drafted with AI assistance and reviewed against our editorial policy before publication. Corrections are made at the source, on the page, with the date shown.