Why AI Observability Needs More Than Just Logs and Prompts

AI-powered applications are reshaping software development, yet many teams still rely on outdated observability practices. A single API call can silently rack up thousands of tokens and hundreds of dollars in hidden costs without raising an alert. Traditional logging—tracking only responses and status codes—misses the real picture: the prompts sent, tools used, reasoning consumed, and financial impact. Without monitoring these four dimensions, teams risk unexpected expenses, undetected failures, and user-facing issues that are nearly impossible to debug. The gap between what’s logged and what’s needed is widening, but new standards and tools aim to close it.

The four signals your AI system must track

Every AI application operates across four critical dimensions that traditional observability tools often overlook. Missing any one of these can leave your system blind to costly or critical failures:

• Logs – Requests, responses, errors, and latency are the foundation, but they only tell part of the story. Traditional APM tools can capture these, yet they fail to distinguish between a successful call and one that was truncated, filtered, or incomplete.

• Prompts – The actual text sent to and received from the model, including system prompts, message history, and tool definitions. Without capturing the full payload, debugging becomes guesswork when user complaints arise.

• Tool calls – The tools an AI agent selects, their arguments, the order of execution, and any retries. Missing this data makes it impossible to diagnose why an agent made incorrect decisions, such as booking the wrong flight.

• Cost – Token consumption (input, output, cached, reasoning) and pricing per million tokens, broken down by user, feature, and request. Without this, teams risk waking up to unexpected budget overruns.

Losing visibility into any of these signals leaves gaps that can cripple an AI system. For example, tracking only logs means missing the true cost of an LLM call, while ignoring tool calls obscures the root cause of agent failures. The challenge isn’t just collecting this data—it’s ensuring it’s comprehensive and actionable.

Logs: why "200 OK" is misleading

A common misconception is that a successful HTTP status code means the AI call worked as intended. In reality, a 200 OK response can mask critical failures:

• A finish_reason of length indicates the response was cut off mid-sentence.
• A content_filter means the output was blocked by safety mechanisms.
• A tool_calls response means the model is waiting for additional work, and the conversation isn’t complete.

Even the most basic log line should include structured data to avoid these blind spots:

• Timestamp, request ID, and parent trace ID for traceability.
• Provider name (e.g., openai, anthropic), model version, and endpoint used.
• Latency metrics, including wall-clock time and time-to-first-token for streaming responses.
• HTTP status, error class, and error body for diagnostics.
• Finish reason to distinguish between truncated, filtered, or completed responses.

Streaming introduces additional complexity. A response may start streaming with an HTTP 200, emit partial content, and then fail silently. Capturing byte count and chunk count helps identify early termination, even if latency-to-first-token appears acceptable. Users perceive speed based on the first token’s arrival, not the total duration—so time-to-first-token is the metric that truly reflects user experience.

Prompts: store the full payload, but redact carefully

Debugging prompt-related issues requires more than just logging the length of a prompt. When a user reports a wrong answer, off-topic response, or unexpected refusal, the only way to diagnose the problem is to examine the exact text the model received. This includes:

• The system prompt.
• Every message in the conversation history.
• Tool definitions and retrieval results embedded in the prompt.

Most homegrown logging systems log prompt.length instead of the actual text, assuming storage is too costly. This assumption backfires when a seemingly minor change in the system prompt causes widespread issues. For example, a stale memory chunk from another user’s session might bleed into a new prompt, leading to irrelevant or incorrect responses. Without the full payload, identifying the root cause is impossible.

While storing full prompts is essential, it introduces privacy risks. Prompts often contain sensitive data like names, emails, addresses, or internal IDs. Shipping raw prompts to third-party observability vendors can turn a debugging tool into a compliance liability. Solutions exist:

• Implement in-network PII redaction before logs leave your system.
• Use tools like Datadog’s LLM Observability, which includes built-in sensitive data scanning for emails and IPs.
• Version system prompts like code artifacts, assigning unique identifiers to each iteration for easier debugging and A/B testing.

A well-structured prompt log might include fields like prompt_version, system_prompt, user_messages, and retrieved_chunks, ensuring every detail is captured without exposing sensitive data.

Tool calls and cost: the hidden layers of AI observability

Tool calls are the actions an AI agent takes beyond generating text. Capturing them requires logging which tools were invoked, their arguments, the sequence of calls, and any retries. Without this data, diagnosing why an agent performed an incorrect action—such as selecting the wrong hotel booking option—becomes impossible. For example, if an agent repeatedly calls a flight search tool but books the wrong flight, the logs must reveal whether the tool was misconfigured or if the model misunderstood the user’s intent.

Cost tracking is equally critical. Traditional logging often misses token consumption details, especially reasoning tokens, which are not always visible in standard API responses. A seemingly small LLM call can silently consume thousands of tokens and hundreds of dollars if reasoning tokens aren’t accounted for. To avoid surprises, logs should include:

• Input, output, cached, and reasoning token counts.
• Per-million-token pricing for each model used.
• Cost breakdowns by user, feature, or request to identify high-spend areas.

In 2026, a standard for capturing these dimensions emerged, but adoption remains slow. Most teams still rely on custom solutions that miss critical fields, leaving gaps in their observability strategy. The good news is that tools and frameworks are evolving to address these challenges, making it easier to track the full lifecycle of AI calls—from prompt to tool execution to cost.

AI observability is no longer just about tracking responses—it’s about understanding the entire lifecycle of an AI call. The days of logging only the output and assuming everything works are over. Teams that invest in capturing logs, prompts, tool calls, and costs will save time, reduce expenses, and deliver more reliable AI-powered experiences. As AI adoption accelerates, the organizations that prioritize observability today will be the ones leading the industry tomorrow.