Observability isn’t just about collecting data; it’s about shaping data so AI can act on it. When we built cortex at airCloset, we found ourselves asking the same question across four critical monitoring surfaces: application code, infrastructure, CI pipelines, and LLM workloads. Each surface needed its own structure to answer the right questions—without overwhelming the AI or the team.
Ryan Tsui, CTO of airCloset, recently outlined this approach in a two-part series. Part 1 focuses on how to design observability for AI by separating four distinct monitoring surfaces, each shaped to answer specific questions. In Part 2, he’ll cover PII handling, integration, and self-healing. The key insight: raw logs and metrics need reshaping before they can be consumed by AI models.
Why raw observability data doesn’t work for AI
The core challenge mirrors what we faced with code search. Throwing 46 repositories of source code at an AI model leads to hallucinations, context overload, and poor results. The same applies to production data.
Raw observability data creates several problems for AI-driven workflows:
- Volume overload: Logs and metrics flood the context window, burying meaningful signals under noise.
- No semantic structure: AI can’t distinguish between errors, warnings, and routine operations without clear labels and relationships.
- Disjointed data: Logs, metrics, and traces live in silos, making it impossible to correlate events across layers.
- Question mismatch: Queries like “What are we spending right now?” can’t be answered by raw logs alone.
To solve this, we adopted a disciplined approach: shape the data for the question it’s meant to answer.
Four observability surfaces, each with a purpose
At cortex, we split observability into four axes, each tailored to answer a specific type of question. This separation ensures clarity for both humans and AI models.
| Monitoring Target | What AI Needs to Answer | Required Shape | |-------------------|-------------------------|----------------| | Application | What’s happening right now in production? | Logs and traces | | Infrastructure | Do we have enough resources or is something down? | Time series metrics | | CI | What broke and since when? | Logs and structured alerts | | LLM | How much are we spending and who’s using what? | Aggregated metrics and structured records |
This isn’t about using different tools—it’s about structuring data so each question gets a clean, fast answer. A single backend trying to serve real-time spending queries alongside historical SQL breakdowns will degrade in performance. Splitting by purpose avoids that trade-off.
Application: Standard OTel stack with strict uniformity
The application layer uses the familiar Grafana stack:
- OpenTelemetry for instrumentation
- Tempo for traces
- Loki for logs
- Mimir for metrics
The magic isn’t in the tools—it’s in the discipline. Every cortex service emits logs and traces in a consistent shape. This uniformity allows AI to run queries like {service_name=""} |~ "error" via MCP and investigate across services without missing context.
We covered application instrumentation in detail in our AI Harness Series, Part 4: Self-Healing. The takeaway: a well-laid OTel stack isn’t just a foundation—it’s the precondition for everything AI-driven that follows.
Infrastructure: Unified metrics for cost and performance
cortex runs on GCP, combining Cloud Run, BigQuery, Pub/Sub, Cloud Tasks, and more. Each resource emits standard metrics—CPU, memory, latency, queue dwell time—which flow into Mimir via Cloud Monitoring.
The real win is centralization. AI can now answer questions like:
- Which service consumed the most CPU last week?
- Is there a worker with a clogged queue?
Because all metrics live in one store, MCP can query them directly. The infrastructure layer doesn’t need custom logic—just clean, consistent data.
CI: Pull-based logs for AI-driven investigations
Most teams rely on GitHub Actions’ UI for CI logs. cortex took a different path: shipping every CI log into Loki.
The rationale is practical:
- Speed: Pull-based access via Loki is faster than hitting GitHub’s API repeatedly.
- Unified search: CI logs sit alongside application logs, enabling cross-querying.
- Structured signals: LogQL alerts convert CI failures into actionable events.
- Natural language queries: AI can ask “Are there tests broken since last week?” and get a precise answer.
The shipping mechanism itself is unconventional. Instead of pushing logs via webhook, cortex performs post-hoc pull. This reduces auth complexity and ensures logs are available for ad-hoc AI queries.
LLM workloads: Structured aggregation for cost and usage
LLM services at cortex require a different shape. We track real-time usage and cost, broken down by team, model, and endpoint. This data lives in a structured store, separate from raw logs.
Why? Because spending questions aren’t about individual requests—they’re about aggregated usage patterns. AI needs clean, pre-aggregated data to answer:
- How much did we spend on LLM calls this hour?
- Which team used the most tokens last month?
This surface is where AI-era design truly surfaces. Without structured aggregation, AI either hallucinates or fails to answer basic cost questions.
Looking ahead: Integration, PII, and self-healing
In Part 2 of this series, we’ll cover:
- How we handle PII safely within observability data
- How these four surfaces integrate into a cohesive AI platform
- How self-healing workflows use this structured observability to act before incidents escalate
Observability for AI isn’t about more tools—it’s about clearer shapes. When each surface answers its intended question cleanly, AI becomes a force multiplier, not a noisy observer.
AI summary
Learn how to shape observability data for AI consumption by separating four monitoring surfaces: app, infra, CI, and LLMs. Avoid raw log overload and enable natural language queries.
