Lab

A One-Off Evaluation Does Not Protect an AI Agent From Drift

Short answer

A successful pre-launch test set describes one version, on selected cases, at one moment. In production, models change, data ages, tools evolve and users invent new uses. Drift is not only a lower score. It is the growing distance between the validated system and the system actually used. ## Five forms of drift

A straight stone path leads to a clear target; beside it, a second path visibly detours around an obstacle. An amber thread marks the honest path, and dull marbles, the failures, sit on a visible shelf.

A successful pre-launch test set describes one version, on selected cases, at one moment. In production, models change, data ages, tools evolve and users invent new uses. Drift is not only a lower score. It is the growing distance between the validated system and the system actually used.

Five forms of drift

Model drift

A new version changes style, policy adherence, tool calls or behavior across languages and rare segments.

Data drift

Sources become stale, structure changes or a new population appears. An answer suited to the old context becomes inadequate.

Workflow drift

Users turn an assistance feature into a decision or execution mechanism. Real authority exceeds designed authority.

Tool drift

An API, permission or integration changes. The agent may continue with less information and no clear signal.

Criterion drift

The definition of a good answer evolves with product, regulation, risk or customer expectations while the evaluation set stays frozen.

Four evaluation rhythms

On every change

Run critical cases when model, prompt, policy, source or tool changes. Block promotion if a safety or authority requirement regresses.

Daily or weekly

Monitor incidents, refusals, corrections, human handoffs, latency and sampled quality. Frequency depends on volume and impact.

Monthly

Review real failures, add representative cases and inspect segments. Compare scores with business outcomes and appeals.

Quarterly

Revisit mission, delegation level, sources, rights, error budget and metric relevance. Remove tests that no longer protect a decision.

Build a living set

Whenever appropriate, turn a useful incident into a regression case with severity, context and expected behavior. Preserve successful cases that protect an important capability as well.

Do not blindly ingest every conversation. Select authorized, minimized and representative examples. Version the set and document why each case was added or removed.

Connect score to action

A threshold should trigger a decision: maintain, investigate, limit, return to assistance or suspend. Separate minor and critical errors, and monitor segments even when the average is stable.

An evaluation that never changes a decision becomes ritual cost.

Preserve traceability

For each run, record system version, dataset version, criteria, evaluator, results, exceptions and decision. Memory should retain the rationale and links to evidence, not every raw output.

FAQ

Is an automated score enough?

No. Automated judges need calibration against human review and real outcomes.

How many cases are required?

Enough to cover material risks and segments. Quality and representativeness matter more than unguided volume.

When should thresholds change?

When authority, use, population or error cost changes, with a documented rationale.

Scope and limits

These are working practices to test in your own context, not a certification or a measured guarantee of reliability. Examples illustrate a possible situation; they are not customer results. A source can be outdated or mistaken, and access controls still need to be checked in the actual system. For a consequential decision, ask an authorized person to review the evidence and record what remains uncertain.

The editorial team is Nawame. Published and reviewed on October 16, 2026; next review: January 15, 2027. This first edition adds navigation and reference links to the supplied article. The references below support the concepts of provenance or risk management; they do not validate Nawame or this proposed practice.

References and reading paths

Where Nawame fits

Nawame can preserve the calendar, reason for each case, evaluated versions, decisions and next review date. Memory becomes the thread from incident to learning to policy change without replacing the evaluation platform or storing every raw output.

Test continuity on your own case.

A ten-minute protocol to check what you retrieve, what is missing and what needs correcting.

Run the diagnostic