Quality becomes measurable

In 30 seconds

AI systems must not only be available. Functional quality, decisions, costs and tool calls must become continuously visible. Classical tests alone are not enough for probabilistic systems.

Why it matters

A non-deterministic system can answer correctly yesterday and wrongly today, without the code having changed. Availability says nothing about functional quality.

Evals, traces down to the tool call, quality SLOs and replay capability turn “it runs” into a robust “it works reliably”.

Editorial assessmentObservability for agents must reach down to the individual tool call. A trace that ends at the model call hides exactly the point where agents do real damage.

The core architecture questions

• How do we measure functional quality — and with which Golden Datasets?
• Are decisions, costs and tool calls traced end to end?
• Which quality SLOs apply, and what happens upon violation?
• Can we reconstruct an incident via replay?
• When does the system escalate to a human?

Typical patterns

• Continuous Evals: ongoing assessment against curated test cases.
• Golden Datasets: stable reference cases for quality comparisons.
• Tool-Call Tracing: traceability down to the individual action.
• Quality SLOs: measurable thresholds instead of gut feeling.
• Replay: reproduce incidents reproducibly.

Common misconceptions & anti-patterns

• Only infrastructure monitoring, no functional quality.
• Tracing ends at the model call instead of the tool call.
• No reference cases — regressions remain unnoticed.
• No defined escalation path to humans.

Practical review questions

1. Do we have Golden Datasets for our most important use cases?
2. Does our tracing reach down to the tool call?
3. Are quality SLOs defined and monitored?
4. Can we fully reproduce a concrete incident?

Link to the end-to-end worked example

How is answer quality tested, how are tool calls logged — and when is a case escalated?

• Golden datasets with typical service cases test the recommendation quality continuously (continuous evals).
• Tracing reaches down to the individual tool call, not just to the model call.
• Quality SLOs and monitored failure patterns trigger defined responses when violated.
• Uncertain or risky cases escalate automatically to a human; incidents can be reconstructed via replay.

To the full worked example: service agent →

Link to the reference architecture

This mission operates in the Control Plane (Evaluation & Quality, Observability & Audit) and is the precondition for being able to operate agents (003) and their identities (004) responsibly at all.

Primary sources

Change history

• 25.06.2026 — Editorial Structured according to the mission template.
• 24.06.2026 — Editorial (subject matter) Initial publication with primary sources.