How to Build a Shadow Workflow That Finds the Right Job for an AI Agent

The most consequential question in agent design is not which model to use. It is where the system should be allowed to act. Process diagrams rarely answer that question. They show the official route through a workflow, while experienced operators rely on exceptions, timing cues, informal policies, and facts scattered across several tools.

A shadow workflow reveals this hidden operating system. The agent receives the same inputs as a human and proposes what it would do, but its actions are never executed. By comparing proposals with real decisions, a team can discover which parts of the job are stable enough to automate, which need approval, and which should remain human.

Step 1: Choose One Decision, Not an Entire Role

Begin with a narrow, repeated decision that has an observable outcome. “Automate customer support” is too broad. “Decide whether an incoming billing request should be refunded, escalated, or answered with policy guidance” is testable.

Write the target as a five-part contract:

• Trigger: the event that starts the work, such as a newly submitted refund request.
• Inputs: the information available at decision time, including the request, transaction record, policy, and account history.
• Allowed proposals: a finite action set, such as approve, deny, request information, or escalate.
• Human decision: the action actually selected by the operator.
• Outcome: later evidence that helps assess the decision, such as a reopened case or supervisor reversal.

Prefer a workflow with frequent examples and reversible consequences. Avoid beginning with irreversible account closures, legal representations, safety decisions, or unusual incidents. Shadow mode prevents execution, but a poorly chosen pilot still teaches little when cases are scarce or outcomes cannot be interpreted.

Common mistake: choosing a task because it consumes time rather than because its decisions can be evaluated. A large administrative burden may conceal several unrelated judgments. Split it until each case has a clear trigger and action set.

Step 2: Capture the Human Decision Trail

The agent and operator must see equivalent evidence. If the human checks an internal dashboard that the agent cannot access, disagreement may reflect missing context rather than weak reasoning.

Create an event record for every case. Preserve the input as it existed when the decision was made; do not reconstruct it from a database that may later change. Record timestamps because available information and policy versions matter.

Ask operators for concise reason codes rather than essays. A controlled set—missing evidence, policy exception, customer risk, system inconsistency—can be analyzed. Permit an “other” field so the taxonomy does not erase discoveries.

Common mistake: treating the human action as unquestionable ground truth. Operators can disagree or apply obsolete policy. The action is a comparison point, not an oracle. Later review should distinguish agent errors from inconsistent human practice.

Step 3: Build the Agent as a Read-Only Proposer

Construct the smallest system capable of making the target proposal. Give it read-only access to the required data and replace every consequential tool with a simulator. A simulated refund tool should validate arguments and log the intended amount, yet never contact the payment system.

Require structured output with fields for proposed action, supporting evidence, missing information, applicable policy, and confidence category. Confidence should be operational rather than theatrical: ready means all required evidence is present; blocked means a named fact is missing; ambiguous means multiple rules plausibly apply.

For example, the agent might return: action “request information”; evidence “transaction exists, delivery state unknown”; missing information “carrier confirmation”; policy “refund-policy-v4, delivery dispute”; confidence “blocked.” This can be audited more reliably than a persuasive paragraph.

Keep the agent invisible to frontline operators during the initial run. If people see its answer before deciding, anchoring contaminates the comparison.

Common mistake: building a fully autonomous architecture for a shadow experiment. Planning loops, numerous tools, and broad memory create extra failure modes. Add complexity only when observed cases demonstrate its necessity.

Step 4: Classify Disagreements Instead of Scoring Matches

A single agreement rate conceals the opportunity. Two systems may choose different actions because the agent missed a fact, the policy is unclear, or the human took an unsupported shortcut. Each cause suggests a different intervention.

Review disagreements using a stable taxonomy:

1. Evidence failure: the necessary fact existed but was not retrieved or noticed.
2. Instruction failure: the agent misunderstood an explicit rule.
3. Policy gap: no written rule resolves the case.
4. Timing mismatch: human and agent acted with different information.
5. Human inconsistency: comparable cases received different treatment.
6. Acceptable plurality: both actions are defensible and materially equivalent.

Suppose the agent approves a refund while the operator requests carrier confirmation. Review shows that a warehouse dashboard contained an unresolved delivery scan, but the agent never received it. This is not evidence that the model cannot reason about refunds. It is an evidence failure and a concrete integration requirement.

Have a domain owner adjudicate a sample of agreements as well. Human and agent can agree for the same wrong reason, particularly when a familiar shortcut violates updated policy.

Common mistake: changing the prompt after every disagreement. This produces brittle instructions and makes results across runs incomparable. Accumulate cases, identify a recurring failure class, then change one component and rerun a fixed evaluation set.

Step 5: Find the Automation Boundary

Do not ask whether the agent is “good enough.” Ask which case conditions produce dependable behavior. Segment results by observable attributes: policy type, missing data, requested value band, account status, language, system involved, and exception code.

A useful pattern may emerge: standard requests with complete transaction data are consistently resolved; delivery disputes fail when carrier state is missing; policy exceptions require judgment. That pattern defines three lanes:

• Automatic lane: execute only cases matching explicit, well-tested conditions.
• Approval lane: prepare an action and evidence packet for human confirmation.
• Human lane: route ambiguous, exceptional, or high-consequence cases without an agent recommendation.

The boundary must be expressible as machine-checkable rules. “Simple cases” is not a boundary. “Policy is identified, all required fields are present, no exception flag exists, and proposed amount matches the deterministic calculation” is.

Common mistake: deploying only the agent’s most confident answers. Model confidence can be poorly calibrated. Gate execution on external conditions and tested case classes, not merely the model’s self-assessment.

Step 6: Turn the Shadow Run Into a Deployment Contract

Before activating any tool, document what the shadow workflow has established. The deployment contract should name eligible cases, prohibited actions, required evidence, approval thresholds, logging obligations, rollback behavior, and the owner who can suspend execution.

Introduce authority gradually. First, show recommendations after humans commit their decisions. Next, let the agent draft actions for approval. Then permit execution within the narrow automatic lane while retaining an immediate kill switch and complete event trail.

Continue sending a portion of eligible cases through independent review. Production conditions drift: policies change, upstream fields disappear, and users learn new ways to phrase requests. Monitor disagreement categories and override reasons, not just completion volume.

Common mistake: treating shadow mode as a one-time launch gate. It is also a permanent instrument. Run it when changing models, prompts, policies, integrations, or action permissions. A new model may improve general reasoning while becoming less reliable on the specific exceptions that matter.

The Concrete Outcome: A Map of Delegable Work

A successful shadow workflow produces more than a model score. It leaves behind a case dataset, an explicit decision taxonomy, a catalogue of hidden information sources, and a machine-readable automation boundary. These assets reveal whether the next investment belongs in model behavior, data access, policy design, or interface redesign.

The deeper opportunity is often unexpected. An agent may not replace the original workflow; it may expose why the workflow was difficult to automate at all. Missing policies can be written. Scattered evidence can be unified. Inconsistent decisions can be standardized. The shadow system thus becomes an instrument of organizational discovery before it becomes an instrument of execution.

This post was drafted with AI assistance and reviewed against our editorial policy before publication. Corrections are made at the source, on the page, with the date shown.