Lab
A One-Off Evaluation Does Not Protect an AI Agent From Drift
Short answer
A successful pre-launch test set describes one version, on selected cases, at one moment. In production, models change, data ages, tools evolve and users invent new uses. Drift is not only a lower score. It is the growing distance between the validated system and the system actually used. ## Five forms of drift

A successful pre-launch test set describes one version, on selected cases, at one moment. In production, models change, data ages, tools evolve and users invent new uses. Drift is not only a lower score. It is the growing distance between the validated system and the system actually used.
Five forms of drift
Model drift
A new version changes style, policy adherence, tool calls or behavior across languages and rare segments.
Data drift
Sources become stale, structure changes or a new population appears. An answer suited to the old context becomes inadequate.
Workflow drift
Users turn an assistance feature into a decision or execution mechanism. Real authority exceeds designed authority.
Tool drift
An API, permission or integration changes. The agent may continue with less information and no clear signal.
Criterion drift
The definition of a good answer evolves with product, regulation, risk or customer expectations while the evaluation set stays frozen.
Four evaluation rhythms
On every change
Run critical cases when model, prompt, policy, source or tool changes. Block promotion if a safety or authority requirement regresses.
Daily or weekly
Monitor incidents, refusals, corrections, human handoffs, latency and sampled quality. Frequency depends on volume and impact.
Monthly
Review real failures, add representative cases and inspect segments. Compare scores with business outcomes and appeals.
Quarterly
Revisit mission, delegation level, sources, rights, error budget and metric relevance. Remove tests that no longer protect a decision.
Build a living set
Whenever appropriate, turn a useful incident into a regression case with severity, context and expected behavior. Preserve successful cases that protect an important capability as well.
Do not blindly ingest every conversation. Select authorized, minimized and representative examples. Version the set and document why each case was added or removed.
Connect score to action
A threshold should trigger a decision: maintain, investigate, limit, return to assistance or suspend. Separate minor and critical errors, and monitor segments even when the average is stable.
An evaluation that never changes a decision becomes ritual cost.
Preserve traceability
For each run, record system version, dataset version, criteria, evaluator, results, exceptions and decision. Memory should retain the rationale and links to evidence, not every raw output.
FAQ
Is an automated score enough?
No. Automated judges need calibration against human review and real outcomes.
How many cases are required?
Enough to cover material risks and segments. Quality and representativeness matter more than unguided volume.
When should thresholds change?
When authority, use, population or error cost changes, with a documented rationale.
Scope and limits
These are working practices to test in your own context, not a certification or a measured guarantee of reliability. Examples illustrate a possible situation; they are not customer results. A source can be outdated or mistaken, and access controls still need to be checked in the actual system. For a consequential decision, ask an authorized person to review the evidence and record what remains uncertain.
The editorial team is Nawame. Published and reviewed on October 16, 2026; next review: January 15, 2027. This first edition adds navigation and reference links to the supplied article. The references below support the concepts of provenance or risk management; they do not validate Nawame or this proposed practice.
References and reading paths
Where Nawame fits
Nawame can preserve the calendar, reason for each case, evaluated versions, decisions and next review date. Memory becomes the thread from incident to learning to policy change without replacing the evaluation platform or storing every raw output.


