How to Build an AI Change Detector That Reads Only What Changed

Most monitoring systems repeatedly ask an AI model to read the same material. A policy page is fetched every morning, a contract is reprocessed after each upload, or a product manual is summarized whenever its version changes. The model spends most of its effort rediscovering what remained constant.

A change detector reverses the unit of work. It first establishes what changed, then gives the model only the altered passage, enough surrounding context to interpret it, and a precise decision to make. The concrete outcome is a pipeline that can watch a set of documents and produce an evidence-backed alert only when a revision affects a defined obligation, risk, deadline, price, or capability.

Step 1: Define a Decision, Not a Topic

“Monitor competitor documentation” is not an operational specification. It does not establish which changes matter or what anyone should do about them. Begin with a decision sentence: Alert the product lead when a competitor adds, removes, or materially changes a documented API capability.

Convert that sentence into explicit impact classes. For API documentation, these might be new capability, removed capability, changed limit, changed authentication requirement, editorial clarification, and irrelevant page furniture. For contracts, they might be altered obligation, deadline, payment term, renewal condition, or governing entity.

For every class, define a destination and required evidence. A changed rate limit might create a product-review task; a corrected spelling should create nothing. An alert should include the old text, new text, source location, detection time, and a concise explanation of impact.

Common mistake: asking the model whether a change is “important.” Importance has no stable meaning without an owner and a decision. Encode the consequence instead: route, record, ignore, or request review.

Step 2: Capture Stable, Comparable Snapshots

Store the source as both raw evidence and normalized content. The raw copy supports audits. The normalized copy makes comparison reliable. A useful snapshot record contains the source identifier, canonical address or file key, retrieval time, source-provided version if available, content hash, raw content, normalized text, and extraction status.

Normalization should remove elements that change without altering meaning: navigation menus, cookie notices, generated timestamps, session identifiers, and repeated footers. Preserve headings, list boundaries, table cells, paragraph order, and meaningful whitespace. Those structures help recover context later.

Give each logical block a stable identifier. A block might be a heading and its paragraphs, a contract clause, a table row, or a configuration object. Match blocks first by native IDs or keys, then by heading path, and finally by textual similarity. This prevents an inserted introductory paragraph from making every later paragraph appear modified.

Common mistake: storing only extracted text. When extraction fails or a classification is challenged, there is no authoritative artifact to inspect. Keep raw snapshots under retention and access rules appropriate to the source.

Step 3: Detect Structural and Semantic Change Separately

Run inexpensive deterministic checks before invoking a model. If the normalized content hash is unchanged, stop. If it differs, compare aligned blocks and label each one as added, removed, modified, or moved. A line diff is useful inside a matched block, but poor as the primary representation because small layout changes can fragment an entire document.

Movement deserves its own label. A clause relocated to another section may retain its meaning, while a clause removed from one agreement and added to another exhibit may alter its scope. Preserve both old and new paths.

Next, filter changes that are textually different but operationally equivalent. Deterministic rules can absorb date-format changes, reordered object keys, or normalized punctuation. Semantic similarity can identify likely paraphrases, but it should not decide impact alone. “May retain data” and “will retain data” are lexically close and operationally distinct.

Consider a worked example. Yesterday’s API page said, “Requests are limited to 100 items per batch.” Today it says, “Requests are limited to 50 items per batch.” The structural layer identifies one modified sentence and the token-level diff isolates the numbers. The semantic layer should mark a changed limit, not summarize the entire page.

Common mistake: sending every detected edit to a language model. This raises cost and creates alert noise. Use exact comparisons and source-specific rules to remove changes whose meaning is already known.

Step 4: Reconstruct the Minimum Sufficient Context

A changed sentence rarely explains itself. “The period is now 30 days” is meaningless without the heading, relevant definition, and preceding sentence. Build a context packet rather than submitting the entire document.

A robust packet contains:

• The old and new versions of the changed block.
• The heading path or clause hierarchy.
• One adjacent block on either side when it clarifies scope.
• Referenced definitions or table headers.
• Source metadata and a stable locator.
• The allowed impact classes and routing rules.

Resolve references selectively. If a clause says “Customer Data as defined in Section 2,” include that definition. Do not attach every defined term. For tables, always include headers; a changed cell value without its column label is unusable.

Set a context budget by priority: changed text first, governing heading second, definitions third, neighboring material last. If the packet exceeds the model’s working budget, split by logical change rather than arbitrary character count.

Common mistake: treating context as a fixed number of characters around the edit. Character windows can sever a definition or combine unrelated sections. Structure is a better boundary than distance.

Step 5: Ask the Model for a Bounded Judgment

The model should classify and explain, not improvise the workflow. Require a structured result containing the impact class, affected entity, old state, new state, recommended route, evidence spans, and a confidence tier. Constrain impact class and route to enumerated values validated by ordinary code.

The instruction should distinguish observation from implication. For the batch-limit example, the observation is that the documented maximum fell from 100 to 50. The implication may be that an integration needs smaller batches. The system should not assert that production is broken unless it has evidence about the integration’s actual behavior.

Use a second pass only for consequential or ambiguous changes. For example, a changed termination clause can be reviewed against the full definitions section, while an obvious spelling correction can be discarded immediately.

Common mistake: letting confidence act as proof. Model confidence is a routing signal, not a guarantee. High-impact changes with incomplete context should still reach a person.

Step 6: Route Alerts With Evidence and Suppression

Build an alert as a compact decision object. Lead with what changed and why it matters. Show old and new excerpts side by side, identify the source and section, then offer the prescribed action. Link to the stored snapshots so a reviewer can inspect the evidence even if the live page changes again.

Prevent duplicate alerts by assigning each change a fingerprint derived from source, stable block ID, old content hash, new content hash, and impact class. If the same revision is fetched repeatedly, update the observation time rather than notifying again.

Group related edits. A release may add an endpoint, authentication scope, and example across three sections. Separate alerts obscure their shared cause. Group by source version and nearby structure, while retaining individual evidence records.

Common mistake: optimizing for immediate notification. A digest is often better for low-impact changes; urgent delivery should be reserved for changes tied to deadlines, access loss, compliance exposure, or production behavior.

Step 7: Test With a Change Corpus Before Expanding

Create a small corpus of paired snapshots with expected outcomes. Include additions, removals, moved sections, paraphrases, numeric changes, negation changes, layout-only edits, extraction failures, and reversions. Record the correct block alignment, impact class, route, and evidence.

Measure the pipeline by stage. Did normalization create a false difference? Did alignment match the wrong blocks? Did context assembly omit a definition? Did classification misread the change? Did routing notify the wrong owner? A single end-to-end accuracy label conceals where repairs belong.

Run the system in shadow mode first: generate records without sending operational alerts. Review false positives and missed changes, then adjust normalization, block matching, and decision rules before changing the prompt. Many apparent model failures begin earlier in the pipeline.

Common mistake: testing only obvious edits. The dangerous cases are small: “not” removed, a threshold reduced, “may” changed to “must,” or an exception inserted under a distant heading.

Step 8: Turn Review Into a Learning Loop

Give reviewers a narrow set of feedback actions: correct, wrong class, wrong impact, insufficient context, duplicate, or missed grouping. Store the correction beside the exact pipeline version, model version, prompt version, and snapshot pair.

Use feedback to repair the appropriate layer. Repeated duplicates call for better fingerprints. Missing table meaning calls for stronger extraction. Incorrect scope calls for reference resolution. Only classification errors supported by adequate context belong in prompt or model work.

Once the pipeline is stable for one decision, add another impact class or source type deliberately. Do not begin by monitoring the entire public web. Begin with one consequential question, one accountable owner, and one source family. The deeper opportunity is not faster reading. It is a system that recognizes when the world has changed enough to require a decision—and remains silent when it has not.

This post was drafted with AI assistance and reviewed against our editorial policy before publication. Corrections are made at the source, on the page, with the date shown.