2025 · ai evaluation architecture
the client wasn't happy. the metrics
didn't show it.
An eval pipeline that scores a production AI agent's conversations and rewrites its prompt when the scores drop. Client satisfaction went up 32% with no new feature code.
built because our QA checks couldn't measure conversation quality
Satisfaction was dropping before any metric showed it. You could see it in the conversations. In staging reviews, the client's replies got shorter and they corrected the agent more often. None of our metrics flagged it. The agent worked, but it kept missing what people were asking for.
A judge agent scores conversations. When scores drop, an optimizer agent rewrites the prompt. The benchmark is a golden dataset of 200+ conversations that people evaluated by hand. The judge scores each conversation against it. When scores drop, the optimizer rewrites the prompt, and a memory layer keeps it from bringing back failures it already fixed. The loop stops once the judge's confidence holds at about 80%. We don't push past that, because overtrained prompts break on real users.
Client satisfaction rose 32% from prompt changes alone. The new prompt handled ambiguous requests, asked clarifying questions, and followed how clients actually talk, which wasn't how the spec assumed they would. As the team shipped new features, the pipeline ran daily so the prompt kept up.
decision log
- 01Benchmark against conversations people gradedThe problem showed up in how users reacted, and none of our metrics measured that.
- 02Stop at about 80% judge confidencePrompts tuned past that point started breaking on real users.
- 03Give the optimizer a memory layerWithout it, a rewrite could bring back a failure an earlier rewrite had fixed.
- 04Run the pipeline dailyThe team kept shipping features, and the prompt had to keep up with them.