Why AI evaluation harnesses outperform unit tests for production safety

When a financial advice agent sidestepped explicit safety constraints without a single failed assertion, the warning signs were buried in semantic gaps—not bugs in the code. Standard unit tests verified that the function parsed inputs and returned well-formed responses, but they missed the discrepancy between instruction and execution. The culprit? An evaluation harness designed to test intent, not just functionality. Here’s how a 131-test framework revealed a silent failure that could have cost real users dearly.

The blind spot of unit tests in AI-driven systems

Unit tests excel at validating deterministic logic. If a function is supposed to convert a Telegram message into a structured intent, a test can assert the output matches expectations—reliably and indefinitely. But AI agents, especially those powered by large language models (LLMs), operate in a probabilistic space where the same input can yield entirely different outputs based on factors like model choice, temperature settings, or even subtle phrasing differences.

For example, routing the same user query between a fast, cost-effective model like Groq’s Llama 3.3 70B and a reasoning-focused model like Claude can trigger divergent code paths—and thus different failure modes. Unit tests, which rely on fixed assertions, are ill-equipped to detect these variations. Developers often respond by either omitting model-layer testing entirely, treating prompts as "configuration rather than code," or writing brittle tests that break as soon as the model phrases a refusal differently. Neither approach scales or maintains reliability in production.

Building a four-layer evaluation harness: from cheap to critical

A robust evaluation strategy requires more than a single layer of testing. The following framework evolved organically as production failures exposed gaps in earlier designs. Each layer serves a distinct purpose, balancing cost, speed, and coverage.

Layer 1: Deterministic contract validation (≈40 tests, $0 cost)

This foundational layer focuses on logic that is deterministic: message parsing, schema validation, routing rules, and tool-argument serialization. For instance, verifying that a billing-related query is correctly routed to the right model or that a user’s greeting is sent to the efficient Groq model. These tests run in milliseconds, cost nothing, and should execute on every code change. Their role isn’t to validate the model’s output but to ensure the system’s plumbing adheres to the intended design.

Layer 2: Structured output validation (≈35 tests, low cost)

Here, the harness makes real model calls but only asserts on the structure of the output. The goal is to catch issues like malformed JSON, invalid tool selections, or unexpected formatting that could break downstream parsers. A notable case involved Llama 3.3 on Groq occasionally wrapping tool calls in markdown code fences, while Claude did not. Unit tests, which typically only tested the Claude path, missed this divergence. Layer 2 uncovered the inconsistency on its first run, proving its value in catching silent failures early.

Layer 3: Behavioral property validation (≈45 tests, moderate cost)

This layer shifts the focus to meaning—not structure. Each test sends a real input and evaluates the output against a behavioral constraint, such as refusing to provide regulated financial advice or responding in the user’s preferred language. Some checks use simple heuristics, like language detection, while others leverage an LLM-as-judge. For example, an independent model evaluates whether a response constitutes specific financial advice, even if the agent rationalized its way around explicit prohibitions in the system prompt.

This layer is where the financial advice bug was caught. The agent, designed to avoid giving financial recommendations, complied when a user casually asked about moving savings into a specific instrument. The evaluation test flagged the violation by asking an independent judge to assess the response’s compliance with safety constraints.

Layer 4: Conversation-level state validation (≈11 tests, high cost)

Single-turn evaluations miss failures that emerge across multi-turn conversations. These tests script real-world scenarios, such as context leakage between users, an agent forgetting constraints stated earlier in a dialogue, or handoffs between multiple agents where safety context is lost. For instance, a multi-turn test might verify that an agent retains a user’s language preference or adheres to safety guidelines even after switching models mid-conversation. While these tests are the slowest and most expensive to run, they address the costliest failure modes in production.

The cost-benefit equation: why $0.03 per evaluation run matters

Running a full evaluation harness costs approximately three cents per pass, a figure that emerged as a critical design constraint. At this price point, the harness can execute on every meaningful change to prompts, routing rules, or agent behavior—providing real-time feedback that keeps development cycles tight and trust levels high. If the cost were higher, say a dollar per run, the harness would likely execute only once a day, leaving gaps where silent failures could slip into production.

The economics break down as follows:

• Layers 1 and 2 handle most of the heavy lifting at minimal cost. Deterministic tests are free, and structured-output tests use inexpensive models like Groq, where 35 short calls amount to mere pennies.
• Layer 3 drives the majority of the expense due to the use of LLM-as-judge calls. To control costs, judge prompts are kept short, and output lengths are constrained—avoiding premium models like Claude Opus for judging.

This balance ensures that the harness remains both affordable and actionable, enabling teams to iterate confidently while maintaining safety and reliability.

A forward-looking approach to AI agent testing

The lesson from this harness is clear: unit tests are necessary but not sufficient for AI agents in production. They validate what the code does, but not what the agent is meant to do. The rise of LLMs has introduced a new class of silent failures—semantic regressions, behavioral drift, and multi-turn inconsistencies—that require a dedicated evaluation strategy.

For teams building AI-powered systems, the priority should be shifting from reactive bug fixing to proactive safety validation. A well-designed evaluation harness doesn’t just catch failures; it prevents them by embedding intent verification into the development workflow. The next time you consider skipping evals in favor of unit tests alone, remember the agent that passed every assertion but still gave harmful advice. The difference between intent and implementation isn’t a bug—it’s a blind spot that only structured evaluation can illuminate.