Definity’s real-time agents cut Spark pipeline failures by 70%

Data engineering teams have long relied on reactive alerts and post-mortems to manage pipeline reliability. But when pipelines power agentic AI systems, even minor failures can derail critical workflows. A new approach from Definity, a Chicago-based startup, shifts this paradigm by embedding intelligent agents directly into Spark pipelines—detecting issues before they escalate.

During its early deployments, Definity helped one enterprise customer identify optimization opportunities within the first week and reduced troubleshooting and optimization effort by 70%. The startup also reports that customers resolve complex Spark problems up to 10 times faster. These gains come as agentic AI systems demand data that is clean, timely, and uninterrupted.

"You need three key elements for agentic data operations: real-time, production-aware context; direct control over the pipeline; and a validation feedback loop," explained Roy Daniel, CEO and co-founder of Definity. "Without all three, you’re observing from the outside—reading a history instead of guiding the present."

Why traditional monitoring falls short at scale

Most pipeline monitoring tools operate externally. Datadog, Databricks system tables, Unravel Data, and Acceldata all collect metrics after a pipeline completes. Even Dynatrace, which participated in Definity’s recent funding round, focuses on post-failure analysis.

Definity’s approach differs fundamentally. By the time external tools detect a problem, the pipeline has already run, the failure has occurred, and the damage—whether wasted compute or corrupted data—is already done.

"It’s always after the fact," Daniel said. "By the time you know something happened, it already happened."

How Definity’s agents work inside pipelines

Definity’s solution places lightweight agents within the pipeline execution layer, not around it. This architectural shift enables proactive intervention rather than retrospective analysis.

• Inline instrumentation. A single line of code installs a JVM agent directly into the Spark execution layer, operating below the platform and collecting telemetry in real time.

• Real-time execution context. The agent monitors query behavior, memory pressure, data skew, shuffle patterns, and infrastructure usage as the job runs. It also dynamically infers data lineage across pipelines and tables—no static catalog required.

• Active intervention. The agent can adjust resource allocation mid-run, halt a job before bad data spreads, or preempt downstream pipelines if upstream sources are stale. In one production example, the agent detected an upstream job preemption and prevented a downstream pipeline from starting, avoiding cascading data corruption.

• Real-time vs. on-demand analysis. Detection and prevention occur in real time, while root cause analysis and optimization recommendations are delivered on demand, packaged with full execution context.

• Minimal overhead, flexible deployment. The agent adds about one second of compute to an hour-long run. Only metadata is transmitted externally; full on-premises deployment is available for environments with strict data residency requirements.

A case study: Nexxen’s shift from reactive to proactive optimization

Nexxen, an ad tech platform running large-scale Spark pipelines on-premises, faced escalating inefficiencies—not pipeline failures—due to a lack of elastic cloud capacity.

"Our biggest challenge wasn’t broken pipelines, but the cost of inefficiency in a fixed infrastructure environment," said Dennis Meyer, Director of Data Engineering at Nexxen. "We had monitoring tools, but they didn’t give us the full-stack visibility we needed to act systematically."

After deploying Definity without modifying any pipeline code, Nexxen identified 33% of its optimization opportunities within the first week. Engineering effort on troubleshooting and optimization dropped by 70%, and the team freed up infrastructure capacity—enabling growth without additional hardware.

"The real shift was moving from reactive troubleshooting to continuous, proactive optimization," Meyer noted. "At scale, the gap isn’t tooling—it’s actionable visibility."

What this means for enterprise data teams

The rise of agentic AI is transforming data pipelines from mere analytics backends into mission-critical infrastructure. Failures that once caused minor delays now block production AI delivery.

For teams running Spark or similar distributed pipelines, the implications are clear:

• Pipeline reliability is now an AI infrastructure problem. Clean, timely data isn’t optional—it’s foundational for AI systems that depend on it.

• Troubleshooting time is a recoverable cost. By catching issues during execution, teams can reduce mean time to resolution (MTTR) and prevent cascading failures.

• Actionable visibility beats raw data. Tools that provide context-rich, real-time insights enable engineers to act before problems compound.

As agentic AI adoption accelerates, the ability to embed intelligence directly into data pipelines may become a competitive differentiator. Definity’s approach suggests a future where pipelines don’t just deliver data—they guard it, optimize it, and ensure it’s ready for the AI systems that rely on it.