Define what good looks like before launch.
A convincing demo is a starting point. For an agentic system, “it completed the task” is not enough. Did it choose the right tool? Retrieve the right information? Interpret it correctly? Recognize when context was missing? Know when to stop, abstain, or involve a person?
Evals make that expected behavior explicit. The evaluation set should reflect real tasks, difficult cases, meaningful failure modes, and situations where autonomous action is not appropriate. Quality thresholds should reflect the consequences of getting an answer or action wrong.
The evaluation method should match the behavior being tested. Some outcomes can be verified deterministically. Some require human review. AI judges can help evaluate subjective outputs at scale, but they also need validation and should not become an unquestioned source of truth.
Monitor the experience people actually get.
Passing an evaluation once does not make an AI system trustworthy forever. Quality, feedback, and failure patterns are product work, starting on day one.
Availability and response time only tell part of the story. Track whether users receive useful, grounded results, whether the system completes the intended task correctly, where users correct or reject its output, and which scenarios repeatedly produce poor outcomes.
Pair automated signals with human review, and define what happens when quality falls below the agreed threshold. Monitoring should tell us more than whether the service is running. It should tell us whether the product is still behaving the way we intended.
Make failures understandable.
Monitoring tells us something changed. Observability gives us the evidence to investigate why: the input, retrieved context, model and prompt versions, tool calls, system actions, and resulting output.
That record needs appropriate access controls and privacy protections. The goal is a traceable path through the system—not a claim that we can inspect a model's private internal reasoning. When a result is challenged, the team should be able to reproduce the conditions and locate where the failure occurred.
Expect the conditions to change.
Even when a model stays the same, the data, tools, user behavior, and tasks around it can shift. Model, prompt, retrieval, and workflow changes introduce additional sources of variation. A strong launch result does not guarantee the same performance later.
Compare production quality against a baseline, review changes across important use cases, and turn new failures into evaluation cases. Decide in advance when to investigate, restrict a capability, route work to a person, or roll back a change.
Use determinism where determinism is better.
Not every decision becomes better because an LLM is involved. Calculations, routing rules, thresholds, eligibility logic, significance scoring, and other well-defined decisions are often better handled deterministically.
I prefer to use AI where ambiguity, synthesis, interpretation, or natural language adds value, while keeping deterministic controls where consistency and auditability matter more.
Someone has to own quality after launch.
AI products often cross organizational boundaries. Product may own the experience. Engineering owns implementation. Data Science or AI Engineering may own models, retrieval, or orchestration. Domain teams own the underlying knowledge.
But someone still has to own the quality of the system as a whole: the evaluation strategy, failure patterns, quality thresholds, monitoring, feedback loops, and decisions about when the product needs to change.
Without that ownership, every component can be functioning as designed while the experience quietly becomes less reliable.
Trust needs evidence, clear ownership, and a way to respond when the system gets it wrong.
Where I've done this
On Landscape iQ, a deterministic scoring framework decides what changed in a payer's coverage, and AI only writes the explanation.
When I took ownership of Searchlight, it included a sophisticated AI summary flow that was good enough for GA but had no path for handling wrong answers. The inputs are formulary documents: hundreds of therapy messages and formats across every payer, all tagged and summarized for changes. At that scale, even a high success rate means some summaries will miss. And unstructured documents are prime territory for drift, because payers change formats and introduce new document types all the time.
So we built for life after launch: a large evaluation set, a process to watch for drift, and a way to correct a single summary manually when a client flags it, instead of paying to rerun the whole document and regenerate every summary in it. The evaluation work also caught a pipeline issue, page-level edits being flagged as coverage changes, before customers saw it. We've raised the quality bar since launch, and we're building the governance to keep evolving the service and catch drift before clients ever notice it.
Further reading: Anthropic on evaluations and production feedback.
