Define what good looks like before launch.
A convincing demo is a starting point. For an agentic system, “it completed the task” is not enough. Did it choose the right tool? Retrieve the right information? Interpret it correctly? Recognize when context was missing? Know when to stop, abstain, or involve a person?
Evals make that expected behavior explicit. The evaluation set should reflect real tasks, difficult cases, meaningful failure modes, and situations where autonomous action is not appropriate. Quality thresholds should reflect the consequences of getting an answer or action wrong.
The evaluation method should match the behavior being tested. Some outcomes can be verified deterministically. Some require human review. AI judges can help evaluate subjective outputs at scale, but they also need validation and should not become an unquestioned source of truth.
Monitor the experience people actually get.
Passing an evaluation once does not make an AI system trustworthy forever. Quality, feedback, and failure patterns are product responsibilities—not postscript engineering tasks.
Availability and response time only tell part of the story. Track whether users receive useful, grounded results, whether the system completes the intended task correctly, where users correct or reject its output, and which scenarios repeatedly produce poor outcomes.
Pair automated signals with human review, and define what happens when quality falls below the agreed threshold. Monitoring should tell us more than whether the service is running. It should tell us whether the product is still behaving the way we intended.
Make failures understandable.
Monitoring tells us something changed. Observability gives us the evidence to investigate why: the input, retrieved context, model and prompt versions, tool calls, system actions, and resulting output.
That record needs appropriate access controls and privacy protections. The goal is a traceable path through the system—not a claim that we can inspect a model's private internal reasoning. When a result is challenged, the team should be able to reproduce the conditions and locate where the failure occurred.
Expect the conditions to change.
Even when a model stays the same, the data, tools, user behavior, and tasks around it can shift. Model, prompt, retrieval, and workflow changes introduce additional sources of variation. A strong launch result does not guarantee the same performance later.
Compare production quality against a baseline, review changes across important use cases, and turn new failures into evaluation cases. Decide in advance when to investigate, restrict a capability, route work to a person, or roll back a change.
Use determinism where determinism is better.
Not every decision becomes better because an LLM is involved. Calculations, routing rules, thresholds, eligibility logic, significance scoring, and other well-defined decisions are often better handled deterministically.
I prefer to use AI where ambiguity, synthesis, interpretation, or natural language adds value, while keeping deterministic controls where consistency and auditability matter more.
Someone has to own quality after launch.
AI products often cross organizational boundaries. Product may own the experience. Engineering owns implementation. Data Science or AI Engineering may own models, retrieval, or orchestration. Domain teams own the underlying knowledge.
But someone still has to own the quality of the system as a whole: the evaluation strategy, failure patterns, quality thresholds, monitoring, feedback loops, and decisions about when the product needs to change.
Without that ownership, every component can be functioning as designed while the experience quietly becomes less reliable.
Trust needs evidence, clear ownership, and a way to respond when the system gets it wrong.
Further reading: Anthropic on evaluations and production feedback.